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PREFACE 


This  paper  was  prepared  by  the  Institute  for  Defense  Analyses  (IDA)  under  the  Task 
Order,  Integrated  Diagnostics,  and  fulfills  an  objective  of  the  task,  to  “provide  analytic 
support  for  a  DoD  sponsored  forum  to  develop  an  in^lementation  approach  for  an  invest¬ 
ment  strategy  for  DoD  automatic  test  systems.”  The  work  was  sponsored  by  the  Director 
of  the  Weapon  Support  Improvement  Group  (WSIG),  Office  of  the  Assistant  Secretary  of 
Defense  for  Production  and  Logistics  (ASD(P&L)). 

On  February  7,  1992,  the  ASD(P&L)  chartered  a  DoD  team  to  develop  a  recom¬ 
mended  DoD  automatic  test  system  investnwnt  strategy,  and  tasked  IDA  to  perform  in- 
depth  technical  and  economic  analyses  in  support  of  this  effort  The  effort  was  directed  by 
a  joint  Executive  Steering  Group  composed  of  personnel  from  the  Office  of  the  Secretary 
of  Defense  (OSD)  and  the  Departments  of  the  Army,  Navy,  and  Air  Force.  The  results  of 
the  IDA  analyses  presented  in  this  paper  were  provided  to  the  DoD  study  team  and  Execu¬ 
tive  Steering  Group,  which  had  the  broader  responsibility  to  consider  recommendations 
and  implementation.  This  pq)er  comprises  two  volumes:  Volume  I,  Sununary  and  Analy¬ 
ses,  and  Volume  n.  Supporting  Data. 

The  overall  study  team  consisted  of  (in  addition  to  the  authors  of  this  paper)  the  fol¬ 
lowing  personnel  from  OSD,  the  Services,  and  IDA:  Ms.  Christine  Fisher  (DoD  Chairper¬ 
son  of  the  Study  Team),  Mr.  Michael  Bloom,  Mr.  Bill  Darling,  Mr.  Steven  Freschi,  CMSgt 
Terry  Lambert,  Mr.  Jonathan  MacMillian,  Mr.  Wesley  McElveen,  Ms.  Kate  O’Sullivan, 
Mr.  Bill  Ross,  CDR  Gus  Scalia,  LCDR  Jim  Seveney,  Mr.  Howard  Sterling,  Maj  Steve 
Topper,  and  Mr.  Perry  Trolinger.  This  team  worked  full  time  for  4  months  and,  along  with 
more  than  80  representatives  of  DoD  and  industry,  participated  in  over  100  hours  of  tech¬ 
nical  interchange  meetings.  Follow-on  activities  of  the  team  continued  for  another  eight 
months,  with  support  provide  by  the  Services  as  required. 

The  following  IDA  research  staff  members  were  reviewers  of  this  document*  Dr. 
Brian  Cohen,  Mr.  Waynard  Devers,  Mr.  Robert  Eberhard,  Dr.  Harlow  Freitag,  Dr.  Richard 
Ivanetich,  and  Dr.  Richard  Wexelblat  Additional  document  reviews  for  techrucal  content 
accuracy,  and  completeness  were  conducted  by  Mr.  Bill  Ross,  U.S.  Navy;  Mr.  Patrick 
Stevens,  U.S.  Army;  and  Mr.  Terry  Lindemann  and  CMSgt  Terry  Lambert  U.S.  Air  Force. 


EXECUTIVE  SUMMARY 


Automatic  test  systems  (ATS)  are  used  to  test  weapon  electronics  in  manufacturing 
and  maintenance.  The  Assistant  Secretary  of  Defense  for  Production  and  Logistics  request¬ 
ed  a  study  to  characterize  the  scope  of  current  ATS  investments  and  to  characterize  mission, 
technical,  and  economic  benefits  of  an  improved  ATS  investment  strategy.  The  Institute  for 
Defense  Analyses  (IDA)  found  that  the  Department  of  Defense  (DoD)  invested  an  estimat¬ 
ed  $35  billion  in  ATS  inventory  from  1980-1992,  with  an  additional  $15  billion  estimated 
for  associated  ATS  support  Based  on  acquisition  programs  and  FY93  Program  Objectives 
Memcnandum  (POM)  funding  levels,  DoD  will  spend  an  estimated  $11  billion  across 
FY93-99  fot  additional  inventory  ATS  development  and  procurement 

Furthermore,  the  analyses  showed  a  strategy  using  common  ATS  to  meet  cross¬ 
weapon  systons  requirements  is  technically  feasible  and  meets  DoD-wide  testing  require¬ 
ments.  Analyses  also  demonstrated  that  diis  strategy  will  reduce  logistics,  improve  opera¬ 
tional  interoperability,  and  significantly  reduce  ATS  acquisition  costs. 

Approach 

In-depth  data  were  collected  on  ATS  supporting  15  weapon  systems  considered  rep¬ 
resentative  of  DoD-wide  programs.  Separate  analyses  were  conducted  to  evaluate  technical 
capabilities  of  modem  ATS,  to  identify  acquisition  and  mission  support  opportunities,  and 
to  evaluate  economic  benefits  of  an  inqnoved  investment  strategy.  Feasibility  assessments 
of  meeting  cross-weqwn  system  and  cross-Service  test  requirements  with  standard  ATS 
families  woe  cartkd  out  based  on  technical  capability  reviews  of  several  current  DoD 
ATS.  ATS  acquisition  and  support  opportunities  were  assessed  based  on  technical  reviews 
of  facttaa  affecting  ATS  technology  development  and  evolution.  A  macro  ATS  investment 
strategy  model  was  devel(^)ed  in  conjunction  with  these  analyses  and  was  used  to  estimate 
ATS  acquisition  costs.  Estimates  of  potential  A'DS  investritent  strategy  savings  were  based 
on  several  analyses:  (1)  the  collection  of  representative  historical  ATS  cost  data  for  the  15 
weiq>on  systems,  (2)  case  study  analyses  that  evaluated  differrat  ATS  investment  strategy 
benefits,  and  (3)  a  macro  assessment  of  DoD  cost  avoidance  opportunities. 


Conduaions 

The  specific  conclusions  arc  addressed  from  three  per^>ectives:  Investment 

Approaches,  Requirements  and  Capabilities,  and  Long-Term  C^portunities. 

Investment  Approaches 

1.  DoD  is  making  duplicative  investments  in  ATS  within  and  across  weapon 
programs  and  depots.  The  majority  of  DoD’s  ATS  are  acquired  separately  to  meet 
individual  user  requirements.  For  the  IS  weapon  systems  reviewed,  over  100  unique 
ATS  types  wore  identified  with  total  test  equipment  numbers  exceeding  2,300.  From 
anotlMT  perspective,  production  quantities  av^ged  less  than  25  ATS  for  each  unique 
ATS  type.  Assessments  show  that  the  technical  testing  capabilities  of  these  unique 
ATS  have  a  high  degree  of  overlap. 

2.  A  strategy  applying  DoD  selected  standard  ATS  families  will  reduce  ATS 
development  and  production  costs  and  provide  operation  and  support  benefits.  A 
maao  ATS  investment  strategy  model  was  developed  as  part  of  this  study  and  used  to 
estimate  DoD  total  ATS  acquisition  costs.  This  model  was  based  on  total  projected 
weapon  system  budgets  and  historical  ratios  of  ATS  acquisition  to  weapon  system 
costs.  Although  "^system-specific”  savings  were  not  addressed  by  this  modeling 
amrroach,  the  analyses  identified  potential  ovoall  ATS  cost  reductions  of  between 
$1.5  and  $1.9  billion  across  FY93-99  weapon  system  acquisition  budgets  based  on 
FY93  POM  funding  projections.  This  assessment  assumed  the  strategy  was 
inqrlemented  at  the  outset  of  FY93.  The  following  factors  were  identified  as  the 
inincipal  causes  for  these  ATS  investment  savings: 

a.  Development  costs  shared  across  multiple  applications, 

b.  Increased  tester  work-loading  efficiency  by  rqplacement  of  multiple  unique 
ATS, 

c.  Economies  of  scale  permitted  by  consolidation  of  ATS  requirements  and  pro¬ 
duction  learning,  and 

d.  Stable  product  base  for  ATS  production,  spares,  and  modifications. 
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Requiraneiits  and  Capabilities 

3.  Existing  ATS  standard  famiiies,  IFTE  and  CASS,  meet  DoD  automatic  testing 
technical  requirements  today  and  will  handle  nearly  all  projected  testing 
requirements  through  the  next  decade  with  only  modest  improvements.  The 
Ai  ny  and  Navy  respectively  have  designated  the  Integrated  Family  of  Test  Equipment 
(IFTE)  and  the  Consolidated  Automated  Support  System  (CASS)  as  their  standard 
families  of  ATS.  Analysis  of  DoD’s  most  modem  ATS  shows  the  CASS  tester  meets 
the  broadest  range  of  testing  capabilities.  No  other  tester  evaluated  covered  the  same 
depth  and  breadth  of  testing  capabilities.  IFTE  was  a  close  second  and  adds  the 
additional  dimension  of  a  highly  mobile  single  ATS.  Both  testers  were  evaluated 
against  potential  future  testing  requirements,  and  analysis  results  showed  that  both 
testers  accommodate  projected  testing  needs  over  the  next  decade  with  only 
modest  ATS  family  improvements.  CASS  and  IFTE  use  industry  standard 
architectures  that  promote  easy  incorporation  of  instruments.  Consequently,  they  may 
be  updated  relatively  easily  to  meet  future  test  needs. 

4.  Appreciable  size  reductitms  of  modem  gmeral  purpose  ATS  are  achievable  only 
by  eliminating  test  system  capabilities  or  by  introducing  highly  tailored  design 
solutions.  Central  to  this  conclusion  are  two  critical  tindings:  (a)  the  frequently  used 
term  “downsized  tester”  addresses  a  concept  of  reducing  ATS  size,  and  this  term  has 
little  meaning  outside  the  context  of  specific  applications  vtdth  existing  ATS,  and  (b) 
tester  downsizing  approaches  are  technology  limited  and  directly  define  tester 
application  limits  by  trading  off  capability  for  smaller  size. 

5.  Built’ln-Test  (BIT)  will  not  substantially  change  ATS  requirements  for  off* 
equipment  repair  over  the  next  decade.  BIT  will  not  eliminate  off-equipment 
(weapon  system)  testing  needs.  BIT  provides  a  valuable  fault  detection  function; 
however,  it  does  not  have  the  same  accuracy  and  range  of  testing  capability  for  fault 
detection,  fault  isolation,  and  repair  verification  as  a  modem  ATS.  In  fact,  current  BIT 
technologies  are  either  limited  or  not  available  for  many  subsystem  testing  domains. 
Analysis  indicated  that  total  ofif-equipimnt  ATS  work-loads  vtdll  not  change 
substantially  because  total  ATS  work-load  requirements  are  determined  by  the 
reliability  of  the  systems  to  be  tested. 
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Long-Term  Opportunities 

6.  Cost-effective  test  software  engineering  and  rehosting  require  new  standards  and 
development  and  support  environments  for  test  information  and  software. 
Although  hardware  represents  the  dominant  research  and  development  (R&D)  and 
production  ATS  costs,  test  software  development,  acquisition,  and  rehosting  still 
represent  significant  costs.  Analyses  indicated  that  the  development  and 
implementation  of  a  common  ATS  software  development  and  support  environment 
would  reduce  total  weapon  system  ATS  acquisition  cost  by  8  to  10%. 

7.  Applying  a  standard  ATS  families  acquisition  strategy  will  not  adversely  affect 
DoD’s  ability  to  acquire  test  resources  from  industry.  DoD  ATS  R&D  and 
production  expenditures  are  spread  across  a  large  number  of  contractors,  with  a  small 
fraction,  typically  less  than  10%,  of  the  ATS  market  in  any  one  firm.  Critical  test 
technical  and  engineering  capabilities  will  continue  to  reside  in  the  commercial  sector. 

8.  Standard  ATS  families  will  promote  the  purposeful  reuse  of  test  equipment  and 
test  software  across  factory,  depot,  and  operational  maintenance.  Standard  general 
purpose  ATS  were  found  to  facilitate  and  promote  the  reuse  of  test  equipment  and  test 
software.  Also,  when  used  at  multiple  maintenance  levels,  common  ATS  families  tend 
to  improve  the  relative  diagnostic  accuracy  of  testing  results. 
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1.  INTRODUCTION 


1.1  PURPOSE 

The  purpose  of  this  study  was  to  detmnine  the  economic  and  technical  factors 
essential  in  the  development  and  support  of  a  Department  of  Defense  (DoD)  automatic  test 
systems  (ATS)  investment  strategy.  The  study,  requested  by  Assistant  Secretary  of  Defense 
for  Production  and  Logistics  (ASD(P&L)),  set  out  to  characterize  the  scale  and  scope  of 
current  ATS  investments  and  to  characterize  economic,  technical,  and  mission  benefits  of 
an  imiHx>ved  ATS  investment  strategy.  Underlying  objectives  of  the  study  were  as  follows: 

•  To  evaluate  ATS  investment  strategy  options  that  meet  both  individual  weapon 
system  and  Joint  Service  needs  in  a  cost-effective  way, 

•  To  characterize  technical  ctqpabilities  of  current  automatic  test  systems,  and 

•  To  project  needed  fiiture  c^bilities. 

1.2  BACKGROUND 

The  DoD  invested  an  estimated  $35  billion  in  ATS  inventory  from  1980  to  1992, 
widi  an  additional  $15  billion  estimated  for  associated  ATS  support  Furthermore,  based 
upon  the  current  planned  acquisition  programs,  DoD  will  spend  an  estimated  $11  billion 
across  FY93-99  for  additional  inventory  ATS  development  and  procurement  If  Defense 
factory  test  equipment  investments  are  included,  the  total  may  be  over  $i00  billion.  Yet  his¬ 
torically  DoD  ATS  acquisition  and  management  activities  have  not  been  weU  coordinated 
across  DoD.  Applied  investment  strategies  have  principally  focused  on  individual  system 
requirements,  often  at  die  expense  of  shared  development  resources,  jiroduction  economies 
of  scale,  and  lost  opportunities  for  common  mission  and  logistics  support  use. 

On  Fetmiary  7,  1992,  the  ASD(P&L)  chartered  a  DoD  team  to  develop  a  recom¬ 
mended  DoD  ATS  investment  strategy,  and  sponsored  the  Institute  for  Defense  Analyses 
(IDA)  to  perform  an  in-depth  technical  and  economic  analysis.  The  effort  was  directed  by 
a  joint  Executive  Steering  Group  composed  of  Office  of  the  Secretary  of  i:>efense  (OSD) 
and  Army,  Navy,  and  Air  Force  personnel.  A  principal  study  team  of  12  people  worked  full 


time  for  4  months,  and  more  than  80  representatives  of  DoD  and  industry  participated  in 
over  100  hours  of  technical  interchange  meetings.  Follow-on  investigation  and  analysis 
continued  with  a  team  ranging  from  four  to  seven  people  for  another  eight  months,  with 
support  provided  by  the  Services  as  required.  Summaries  of  the  data  collected  and  results 
of  the  IDA  technical  and  economic  analyses  are  presented  in  this  paper. 

1.3  GLOSSARY 

Automatic  Test  System  (ATS):  Automatic  test  system  is  defined  to  include  the 
automatic  test  equipment  (ATE)  hardware  and  operating  software,  test  program  sets  (TPS) 
that  include  the  hardware  connectors  and  software  programs  to  test  individual  weapon  elec¬ 
tronic  items,  and  the  associated  ATS  software  development  environments.  Figure  1  illus¬ 
trates  the  principal  ATS  elements  and  includes  listings  of  sample  functions. 

Automatic  Test  Equipment  (ATE):  Automatic  test  equipment  includes  the  oper¬ 
ating  system  or  executive  software  that  runs  on  the  main  computer  and  a  range  of  hardware 
components.  ATE  hardware  components  consist  of  items  such  as  the  main  computer  within 
the  test  equipment,  operating  system,  stimulus  and  measurement  instruments,  signal  con¬ 
trol  and  switches,  and  needed  interfaces  such  as  heating  and  cooling  sources  and  structural 
supports. 

Test  Program  Sets  (TPS):  Test  program  sets  consist  of  interface  test  adapters 
(interface  devices),  test  program  software,  and  test  program  documentation  and  data.  The 
test  program  set  provides  necessary  resources  and  information  to  test  an  item  on  an  auto¬ 
matic  test  equipment  Development  and  production  of  a  specific  interface  from  automatic 
test  equipment  to  the  unit  under  test  (UUT)  are  performed  as  part  of  test  program  set  devel¬ 
opment  and  production.  Test  programs  are  generally  written  for  each  unit  under  test  The 
test  program  set  must  provide  all  additional  resources  not  available  in  the  designated  auto¬ 
matic  test  equipment,  but  which  are  necessary  to  test  the  unit  under  test 

ATS  Software  Development  Environment:  The  test  program  set  development 
environment  includes  the  descriptions  of  the  ATE  and  related  equipment  architectures  and 
interfaces,  programming  and  test  specification  languages,  compiler,  and  provisions  for  cap¬ 
turing  and  using  design  test  requirement  and  test  strategy  information  concerning  the  unit 
to  be  tested. 

General  Purpose  Automatic  Test  Equipment:  Automatic  test  equipment  is  gen¬ 
eral  purpose  when  it  is  designed  to  meet  a  broad  spectrum  of  electronics  testing  (direct  cur¬ 
rent  (DC)  to  light,  large  frequency  ranges,  laige  power  ranges,  etc.).  Furthermore,  to  be 
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Figure  1.  Automatic  Test  System  (ATS)  Principal  Elements 


general  purpose,  the  automatic  test  equipment  must  be  capable  of  testing  multiple  types  of 
full  systems  and  subsystems  including  black-boxes  and  circuit  cards,  and  must  provide 
fault  detection  and  fault  isolation  capabilities  down  to  a  board  and  component  level. 

Ai  S  Applications:  DoD  automatic  test  systems  are  used  to  test  weapon  system 
repairable  items  by  aiding  in  the  tasks  of  (1)  identifying  the  replaceable  component,  (2) 
adjusting  a  repairable  item  to  technical  order  specifications,  and  (3)  assuring  a  repaired  item 
is  ready  for  issue.  Automatic  test  systems  are  used  in  DoD  field  and  depot  electronics  main¬ 
tenance  environments  to  reduce  troubleshooting  times,  to  augment  the  skills  of  technicians, 
and  to  test  electronics  technologies  that  are  difficult  or  impossible  to  test  manually.  Auto¬ 
matic  test  systems  are  also  used  in  manufacturing  for  in-process  and  acceptance  testing  of 
electronics  related  products. 

DoD  Standard  ATS  Family:  V^ithin  the  context  of  this  paper,  a  DoD  standard  ATS 
family  is  an  automatic  test  system  with  general  purpose  capabilities  that  meet  the  testing 
needs  of  multiple  DoD  systems,  and  has  been  designated  by  one  or  more  of  the  Services  as 
a  common  automatic  test  system  for  multiple  weapon  system  testing  applications.  >^thin 
this  context,  DoD  has  identified  two  existing  Al^  standard  families:  the  Army  Integrated 
Family  of  Test  Equipment  (IFTE)  Base  Shop  Test  Facility  (referred  to  as  iFl  H  throughout 
this  paper  unless  an  exception  is  ;^pecifically  cited),  and  the  Navy  Consolidated  Automated 
Support  System  (CASS).  Both  of  these  automatic  test  systems  were  proposed  for  designa¬ 
tion  as  initial  DoD  standard  ATS  families  for  immediate  application  throughout  DoD.  A 
formal  process  for  identifying  requirements  and  adding  other  automatic  test  systems  to  this 
designated  list  of  DoD  standard  ATS  families  was  not  established  during  the  study  period. 
Other  potential  family  candidates  include  commercial  testers  used  for  some  depot  applica¬ 
tions,  testers  for  special  application  areas  such  as  electronic  warfare,  and  DoD-developed 
automatic  test  systems. 
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1.4  REPORT  ORGANIZATION 


This  report  is  arranged  as  four  major  sections  (Parts  I,  n.  III,  and  IV)  which  have 
been  divided  into  two  volumes.  The  first  three  sections  are  in  Volume  I,  and  the  last  section, 
along  with  the  References,  Bibliography,  and  List  of  Acronyms,  are  in  Volume  II. 

VOLUME  I:  Summary  and  Analyses 

a.  Part  1:  Introduction  provides  a  very  brief  introduction,  including  purpose,  report 
organization,  glossary,  and  background. 

b.  Part  H:  Conclusions  provides  a  short  summary  list  of  the  primary  conclusions 
and  follows  with  a  more  detailed  discussion  of  each. 

c.  Part  ni;  Analyses  provides  summaries  of  data  collected  during  the  study  and 
analyses  that  support  the  primary  conclusions  presented  in  Part  D.  Secondary 
findings  and  conclusions  are  also  documented  within  sections  that  discuss  the 
following: 

1 .  baseline  data  analysis, 

2.  ATS  investment  strategy  option  case  study  analysis, 

3.  DoD  ATS  investment  analysis, 

4.  analysis  of  DoD  and  Service  ATS  management  policies  and  organizations, 

5.  assessment  of  ATS  requirements  and  applications,  and 

6.  assessment  of  ATS  technology  development  and  evolution. 

VOLUME  II:  Supporting  Data 

d.  Part  IV  includes  weapon  system  profiles,  ATS  baseline  data  summaries, 
sununaries  of  selected  ATE  comparisons,  definitions  of  ATS  investment 
strategy  options,  and  lists  of  study  participants. 
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PARXn 

CONCLUSIONS 


9 


2.  DISCUSSION  OF  CONCLUSIONS 


DoD- wide  use  of  standard  ATS  families  provides  the  best  opportunity  to  meet  auto¬ 
matic  testing  needs  at  the  lowest  possible  cost  The  study  investigated  automatic  test  sys¬ 
tems  principally  used  to  test  weapon  system  electronics  related  products.  Analyses  revealed 
significant  acquisition  cost  avoidance,  and  logistics  savings  and  improventents  are  possible 
if  DoD  components  meet  their  ATS  needs  by  selecting  from  designated  standard  ATS  fam¬ 
ilies.  The  eight  specific  conclusions  address  ATS  from  three  perspectives:  Investment 
Approaches,  Requirements  and  OqrabUities,  and  Long-Term  Opportunities.  These  conclu¬ 
sions  are  discussed  in  greater  detail,  with  supporting  findings,  in  the  following  sections  of 
this  chapter.  The  in-depth  analyses  from  which  these  conclusions  are  drawn  are  presented 
in  Part  m.  Sections  3  dirough  9. 

Investment  Approaches 

a.  DoD  is  making  duplicative  investments  in  ATS  within  and  across  weapon  pro¬ 
grams  and  depots. 

b.  A  strategy  iqiplying  DoD  selected  standard  ATS  families  wiU  reduce  ATS 
development  and  production  costs  and  provide  operation  and  support  benefits. 

Requironents  and  Capabilities 

c.  Existing  ATS  standard  families,  IFTE  and  CASS,  meet  DoD  automatic  testing 
technical  requirements  today  and  will  handle  nearly  all  projected  testing 
requirements  through  the  next  decade  widi  only  modest  improvements. 

d.  Appreciable  size  reductions  of  modem  general  purpose  ATS  are  achievable 
only  by  eliminating  test  system  capalnlities  or  by  introducing  highly  tailored 
design  solutions. 

e.  Built-In-Test  (BIT)  wUl  not  substantially  change  ATS  requirements  for  off- 
equipment  rqiair  over  the  next  decade. 
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Long'Term  Opportunities 

f.  Cost-effective  test  software  engineering  and  rehosting  requires  new  standards, 
and  development  and  support  environments  for  test  information  and  software. 

g.  Applying  a  standard  ATS  families  acquisition  strategy  will  not  adversely  affect 
DoD’s  ability  to  acquire  test  resources  from  industry. 

h.  Standard  ATS  families  will  promote  the  purposeful  reuse  of  test  equipment  and 
test  software  across  factory,  depot,  and  operational  maintenance. 

2.1  DOD  DUPLICATIVE  ATS  INVESTMENTS 

DoD  is  making  duplicative  investments  in  automatic  test  systems  within  and 

across  weapon  programs  and  depots. 

2.1.1  ATS  Duplication 

The  development  costs  for  standard  automatic  test  systems  which  nneet  multiple 
applications  are  typically  the  same  as  those  for  a  single  weapon  ATS.  Figure  2  compares 
ATE  development  cost  profiles  for  a  limited  application  and  two  multi-application  automat¬ 
ic  test  systems  over  their  respective  lifecycles.  Large  repeated  development  investments  for 
each  single  .system  application  go  principally  for  hardware  and  software  integration:  con¬ 
sequently  the  same  testing  capability-  is  duplicated  on  each  new  automatic  test  system.  This 
is  illustrated  by  the  fact  that  large  percentages  of  current  ATS  are  using  previously  devel¬ 
oped,  commerciaUy  available,  components  (e.g.,  for  Consolidated  Automated  Support  Sys¬ 
tem  (CASS)  and  Integrated  Family  of  Test  Equipment  (IFTE)  approximately  65%)  many 
of  which  provide  the  same  capability.  These  duplicative  investments  also  extend  over  the 
ATS  life  cycle. 

The  F-16  Avionics  Intermediate  Shop  (AIS)  is  a  weapon-unique  intermediate  main¬ 
tenance  ATE  that  was  developed  specifically  for  die  F-16  aircraft  The  F-16  AIS  initial 
development  costs,  at  $178  million,  iq)proached  CASS  development  costs  at  $220  million 
(both  FY93$).  However,  CASS  is  currently  targeted  for  some  30  weapon  systems/sub¬ 
systems,  while  the  initial  F-16  AIS  only  supported  the  avionics  of  the  A/B  series  F-16  air¬ 
craft  IFTE  initial  development  costs  were  $70  million;  present  IFTE  fielding  plans  are  in 
place  for  14  weapon  systems/subsystems.  While  the  $70  million  accurately  reflects  the 
Army’s  initial  development  funding  for  IFTE,  additional  development  resources,  both 
Government  and  contractor,  were  expended  against  the  IFTE  concept  when  Grumman  was 
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Cnnalativc  R&D  Cost  Profile 

I  Assumes  Mid-Life  Update  for  CASS  &  DTE  at  10  Years 

Assumes  2  Additional  Weapon  System  Testing  Requirements  Added  Each  Year 

(1  CASS  &  1  DTE) 


Figure  2.  Development  Cost  Profile  Comparisons  of  a  System  Unique  (F*16)  and 
Two  Multiple  System  Application  ATE  (CASS  &  IFTE) 

in  head-to-head  competition  with  General  Electric  to  win  award  of  the  CASS  development 
and  production  contracts. 

These  initial  ATS  developments  are  followed  by  longer  term  developments  of  ATS 
improvements  and  updates  that  usually  approach  the  initial  costs.  Automatic  test  systems 
require  system  maturation  and  technology  modifications  over  their  lifecycle.  For  example, 
automatic  test  systems,  historically,  experience  mid-life  upgrade  if  employed  for  more  than 
8  to  10  years.  In  the  case  of  the  F-16,  later  AlSs  that  were  design  evolutions  of  the  original 
A/B  AIS  automatic  test  equipment  supported  the  avionics  of  the  C/D  scries  aircraft  The  F- 
16  AIS  automatic  test  equipment  over  its  life  required  a  total  dcvelopmentAnodification 
design  investment  of  nearly  $700  million  (FY93$).  The  same  development/modification 
design  investment  (approximately  $700  million)  is  projected  for  CASS  over  a  similar 
lifespan.  However,  in  contrast  to  the  F-16  AIS  series  of  testers,  CASS  may  support  as  many 
as  45  different  weapon  systems/subsystems  (die  original  30  plus  the  addition  of  1  new 
weapon  system  per  year  as  illustrated  in  Figure  2). 
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Analyses  revealed  a  proliferation  of  ATS  types,  many  of  which  had  duplicative 
capabilities.  For  example,  47  unique  automatic  test  systems  (plus  1  additional  developntent 
under  current  consideration)  were  identified  for  the  F-18,  M-1,  F-16  and  F-15  weapon  sys-  • 

terns.  The  47  ATS  types  were  spUt  between  field  (intermediate  or  “I”)  and  depot  level  appli¬ 
cations. 


For  the  15  weapon  systems  reviewed,  over  100  unique  ATE  types  were  identified 
with  total  equipment  numbers  exceeding  2,300.  Table  1  summarizes  the  types  and  quanti¬ 
ties  observed.  Since  the  study  focused  on  direct  weapon  system  electronics  test  equipment, 
other  related  weapon  system  items  (e.g.,  munitions,  electronic  warfare  (EW)  pods)  are  not 
included  in  these  tables.  ATS  experts  at  technical  interchange  meetings  (TIMs)  concluded 
that  there  existed,  within  the  operational  life  of  each  ATS  type  identified  by  weapon  system 
in  Table  1,  multiple  general  purpose  automatic  test  equipment  that  met  or  exceeded  95%  of 
the  technical  testing  requirements  of  these  systems.  In  odier  words,  the  ATE  capabilities 
such  as  measurement,  stimulus,  timing,  and  loading  would  accommodate  the  test  needs  of 
the  weapon  system  in  excess  of  95%  of  all  specified  technical  testing  requirements.  These 
observations  are  supported  for  current  generations  of  general  purpose  testers  by  detailed 
comparisons  of  testing  capabilities  summarized  in  Appendix  D,  ATE  Comparisons. 


Table  1.  Summary  of  ATE  Types  and  Quantities  by  Weapon  Systems 


NAVY 

ARMY 

AIR  FORCE 

Weapon 

ATS 

Weapon 

ATS 

Weapon 

ATS 

System 

Types 

Qty 

System 

Types 

Qty 

System 

Types 

Qty 

A-12 

1 

62 

ABRAMS 

4 

202 

ACM 

4 

16 

AMRAAM  * 

2 

27 

BRADLEY 

2 

230 

C.17 

2 

12 

F-18 

9 

453 

APACHE 

6 

47 

F-IS 

24 

402 

1 

10 

MLRS 

5 

14 

F-16 

10 

444 

SQQ-89 

5 

14 

AVENGER 

3 

48 

F-22  (est) 

25+ 

338 

*  Jmnt  Air  Force/Navy  program.  Air  Force  is  lead  service;  however.  Navy  cdlected  data  for  this  study. 

Examples  of  ATS  proliferation  and  duplicative  development  and  acquisition  are 
illustrated  by  the  many  F-18  automatic  test  systems.  If  CASS  had  been  available  at  die  out¬ 
set  of  the  F- 1 8  program,  the  requirements  for  over  350  automatic  test  systems  of  8  different  • 

types  could  have  been  met  with  only  64  CASS  stations.  In  another  example,  two  Services 
test  the  same  prime  equipment,  yet  use  different  automatic  test  equipment  (e.g.,  the  Air 
Force  and  Navy  use  different  ATS  to  test  the  AIM-9L/M  Sidewinder  Missile  Guidance  and 
Control  Section).  # 
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Hiese  findings  of  duplicative  investments  correlate  with  the  findings  of  earlier  stud¬ 
ies  reviewed  in  the  course  of  this  investigation.  ATS  proliferation  and  duplicative  develop¬ 
ments  documented  in  a  study  of  automatic  test  equipment  sponsored  by  the  Secretary  of  the 
Navy  in  1976,  lead  to  the  cost  benefit  analysis  (CBA)  and  competitive  development  of 
CASS.  In  the  past  year,  DoD  ATS  management  has  been  the  subject  of  4  DoD  Inspector 
General  (IG)  audits,  three  General  Accounting  Office  (GAO)  audits,  and  most  recently  a 
House  Armed  Services  Committee  (HAC)  special  investigation.  These  have  all  reflected 
the  lack  of  a  DoD  policy  or  coordinated  approach  to  automatic  test  systems  (to  be  discussed 
in  the  next  section),  and  have  criticized  the  resulting  ATS  proliferation. 

2.1J1  Decentralized  ATS  Management 

Decentralized  ATS  management,  within  and  across  Service  and  functional  lines,  is 
viewed  as  a  major  contributing  factor  to  duplicative  ATS  investments.  Historically,  DoD 
ATS  investment  decisions  have  been  made  predominately  in  a  decentralized  fashion  by 
individual  weapon  program  and  depot  managers. 

The  Army  and  Navy  have  begun  to  implement  centralized  ATS  management  strat¬ 
egies  and  organizations,  and  the  Air  Force  has  a  dispersed  management  strategy  giving  ATS 
acquisition  process  and  standardization  implementation  authority  to  individual  weapon 
system  program  offices.  The  Army  and  Navy  are  making  progress  with  partial  implemen¬ 
tation  of  Service-level  polices  to  use  ATS  families.  The  Air  Force,  during  the  course  of  this 
study,  was  undergoing  a  reorganization  that  combines  the  Systems  and  Logistics  Com¬ 
mands.  In  addition,  at  the  outset  of  this  study  the  Air  Force  was  proposing  to  establish  an 
ATS  management  office  identified  as  an  Integrated  Weapon  System  Management  (IWSM) 
office;  and,  as  the  study  was  concluding,  the  Air  Force  formalized  its  ATS  management 
approach  by  establishing  an  ATS  Product  Group  Manager  (PGM). 

Even  with  current  Service  management  initiatives  targeted  towards  improving  ATS 
acquisition  processes,  approximately  70%  of  all  ATS  fimds  are  ^nt  on  unique  ATS 
requirements.  Opportunities  to  consolidate  requirements  and  reduce  the  percentage  of 
unique,  and  often  duplicative,  ATS  are  lost  due  to  the  decentralized  ATS  policy  and  decen¬ 
tralized  management  applied  across  the  Department  of  Defense.  Table  2  summarizes  the 
decentralized  conditions  found  during  this  study. 

The  Army  and  Navy  have  several  early  generations  of  ATS  families  (e.g.,  EQUATE 
&  Versatile  Avionics  Shop  Test  (VAST)  System),  with  their  most  current  being  IFTE  and 
CASS  respectively.  Partial  centralized  management  implementations  for  both  the  Army 
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Table!.  ATS  Policy  and  Management  Summary 


DoD 

Army 

Navy 

Air  Force* 

Polky 

None 

«* 

Service  Level 

Service  Level 

SystemAxrgiaics 

Command 

Level 

AcquisitioB 
Control  & 
Standardiation 

ATLAS 

Common  Family 

(IFTE) 

Common  Family 

(CASS) 

Standard 
SpecifK:ation  Set 
(MATE  Guides) 

TPS 

Deveiopnient& 

Process 

Control 

None 

Varies  by  Program 

(AMC 

Guidance) 

CASS-Relaled 

Control 

(NAVAIR 

Mgmt) 

Varies  by 
Program 

(MATE  Guides) 

ATS 

R&D 

Organization 

None 

Service  Wide 

Limited  Serviced 
CASS  Wide 

Program  Specific 
&MATEStds 

ATS 

Management 

Organization 

None 

Service  Wide 

(PM  TMDE) 

CASS  Wide 

(NAVAIR) 

Program  Specific 
&MATEStds 

(ProgOffi'ALC) 

*  Air  F(Hx:e  proposing  changes  (reflects  status  at  time  of  study). 

**  DoD  Directive  4151.18,  Maintenance  of  Military  Materi^,  was  published  August  12,  1992 
(requires  ATS  be  standardized  among  the  DoD  components). 

***  Abbreviated  Test  Language  for  All  Systuns  (ATLAS)  allowed  in  lieu  of  Ada  {HOgiamming 
language  by  DoD  Instruction  5000.2. 

Acronyms  not  laeviously  defined:  Air  Logistics  Center  ( ALC);  Army  Material  Command  ( AMC); 
Moduto  Automatic  Test  Equipment  (MATE);  Naval  Air  Systems  Command  (NAVAIR);  Program 
Manager,  Test,  Measurement,  and  Diagnostic  Equipment  (PM  TMDE). 

and  Navy  were  considered  primary  motivating  forces  for  common  ATS  acquisitions.  Naval 
Air  Systems  Command  (NAVAIR)  reduced  duplicative  ATS  investments  with  a  central 
organization  that  consolidates  common  requirements  and  then  funds  conunon  needs  with 
budget  category  Aircraft  Procurement  Navy  (APN)-7  funding  for  conunon  support  equip¬ 
ment.  This  funding  policy  approach  applies  to  NAVAIR,  but  not  to  surface  and  submarine 
programs.  No  similar  funding  approach  was  observed  elsewhere  in  the  Navy.  The  Army 
Program  Manager,  Test,  Measurement,  and  Diagnostic  Equipment  (PM  TMDE)  authority 
is  invoked  by  a  waiver  process.  Army  study  pertonnel  reported  that  waivers  are  often  sub¬ 
mitted  late  in  the  acquisition  process,  and  the  study  found  that  all  early  automatic  test  sys¬ 
tem  decision  authority  resides  with  individual  weapon  system  program  offices. 

The  Air  Force  delegates  decisions  to  dispersed  weapon  and  depot  program  manag¬ 
ers.  The  Modular  Automatic  Test  Equipment  (MATE)  Program  was  the  Air  Force's  attempt 
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to  reduce  proliferation  and  reduce  ATS  costs.  The  failure  of  MATE  to  have  a  significant 
influence  on  these  two  areas  (ATS  proliferation  and  costs)  was  attributed  to  a  lack  of  imple¬ 
mentation  authority  and  support  at  the  weapon  system  program  manager  and  weapon  sys¬ 
tem  prime  contractor.  Of  323  potential  programs,  less  than  half  (126)  followed  MATE 
procedures,  only  45  were  MATE  baselined  programs,  and  only  about  half  of  this  number 
went  on  contract  (24).  At  present  the  Air  Force  is  reported  to  be  pursuing  new  ATS  acqui¬ 
sition  programs  for  the  F-22,  the  F-IS  (downsized  tester),  and  the  F-16  (downsized  inter¬ 
mediate  tester).  None  of  these  new  acquisition  programs  require  use  of  the  same  automatic 
test  system;  yet  the  bulk  of  the  development  work  involves  the  duplicative  integration  of 
already  existing  capabilities:  stimulus,  measurement,  signal  switching,  interface  buses,  and 
general  purpose  computing. 

Finally,  no  DoD-level  policy  nor  management  structure  limiting  ATS  acquisition 
decisions  was  identified  during  the  study.  The  lack  of  mechanism  and  policy  to  foster  cross- 
Service  ATS  applications  was  found  to  be  a  major  impediment  to  consolidating  DoD-wide 
ATS  requirements  and  investments.  Reduction  of  duplicative  investments  can  be  achieved 
more  effectively  with  DoD-level  ATS  acquisition  policy  and  centralized  ATS  nuuiagement 
structures.  Use  of  the  common  ATS  families  across  systems  does  not  limit  a  weapon  system 
program  manager’s  flexibility  to  acquire  and  support  unique  test  programs  to  meet  weapon 
system  specific  test  requirements  and  mission  profiles. 

2,2  AN  IMPROVED  ATS  ACQUISITION  STRATEGY 

A  strategy  t^yplying  DoD  selected  standard  ATS  families  will  reduce  ATS 
development  and  production  costs  and  provide  operation  and  support 
benefits. 

2,2.1  Developinent  and  Productitm  Costs 

Case  study  results  showed  that  a  DoD  acquisition  strategy  based  upon  wide  appli¬ 
cation  of  standard  general  purpose  ATS  families  will  reduce  total  DoD  expenditures. 
Extending  this  analysis  across  projected  DoD  weapon  system  acquisition  budgets  revealed 
cost  avoidance  oj^ortuiuties  that  ranged  into  the  billions  of  dollars. 

The  case  study  analysis,  based  on  historical  DoD  ATS  costs  data,  compared  invest¬ 
ment  strategy  options: 

•  No  change  in  strategy  •  Adopting  common  ATS  families 

•  Adopting  common  ATS  specifica-  •  Adopting  common  software  envi- 

tions  ronment 
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The  cax  studies  evaluated  the  relative  ATS  investment  expenditures  (development 
and  production)  for  the  F-18,  M-1,  F-16,  and  F-IS  weapon  systems  over  a  20- year  period. 
Analysis  results  showed  that  a  standard  ATS  family  acquisition  approach  could  produce  an 
average  ATS  acquisition  savings  of  2S  to  35%  or  more  for  each  weapon  system.  The  poten¬ 
tial  DoD  ATS  acquisition  savings  that  could  have  been  realized  for  these  systems  alone  was 
conservatively  estimated  in  excess  of  $2.35  billion.  The  common  ATS  family  investment 
strategy  option  produced  the  greatest  savings  for  each  of  the  weapon  system  case  studies, 
yet  additional  savings  of  lesser  magnitude  wae  possible  with  the  common  ATS  specihca- 
tions  and  common  software  environment  strategies  (more  details  follow  in  Section  5).  Case 
study  sensitivity  analyses  of  the  common  ATS  family  acquisition  strategy  approach 
revealed  that  less  conservative  assessments  produced  estimates  of  even  greater  savings.  For 
example,  a  less  conservative  analysis  based  on  tite  F-16  ATS  case  study  identified  addition¬ 
al  savings  of  $373  million  over  the  $2.35  billion. 

The  analysis  was  extended  beyond  these  4  weapon  system  case  studies  to  the  full 
DoD  weapon  system  acquisition  budget  The  evaluated  investment  strategy  was  a  compos¬ 
ite  of  the  ATS  common  family  and  the  common  software  environment  strategies,  with  the 
principal  benefits  coming  from  the  family  approach.  This  extended  analysis  across  project¬ 
ed  DoD  weapon  system  acquisition  budgets  identified  ATS  cost  avoidance  opportunities  of 
1.5  to  1.9  billion  dollars,  estimated  for  a  period  from  FY93  through  FY99.  The  assessment 
was  based  on  the  assumption  that  the  strategy  had  been  implemented  at  the  beginning  of 
the  FY93  budget  year.  These  potential  savings  are  summarized  in  Table  3. 


Table  3.  Potential  ATS  Acquisition  Strategy  Savings 


Average  Per  Weapon  System 

DoD  Wide 

ATS  Acquisition 
(Development  & 
Production)  Savings 

25%  to  35% 

$1.5  to  1.9  Billion 
(over  6- Year  Period) 

Reasons 

•  Share  Development  Costs 

•  Share  Production  Overhead  (Economies  of  Scale) 

•  Benefit  from  Production  Learning 

•  Increase  Tester  Work-Load  Efficiency 

•  Benefit  from  Stable  Product  Base 

The  following  paragraphs  help  to  illustrate  several  of  the  reasons  presented  in  Table 
3.  The  benefits  of  shared  development  costs  are  presented  as  Research  and  Development 


(R&D)  cost  avoidance.  The  benefits  of  reduced  ATS  type  proliferation  are  presented  as  pro¬ 
duction  unit  cost  reductions.  The  benefits  of  increased  tester  work-load  efficiencies  are  pre¬ 
sented  as  reduced  ATS  production  quantities.  More  details  follow  in  Section  6. 

R&D  Cost  Avoidance.  A  DoD  ATS  standard  family  strategy  avoids  duplicative 
development  efforts  because  the  general  purpose  testing  capabilities  of  existing  ATS  fanfuly 
members  meet  multiple  system  applications.  Comparisons  of  weapon  system  testing  needs 
to  the  general  purpose  testing  capabilities  of  CASS  and  IFTE  showed  these  ATE  can,  with 
minimal  modifications,  meet  most  (over  95%  on  average)  of  DoD’s  technical  testing 
requirements  for  the  next  ten  years.  There  is  virtually  no  need  for  new  unique  ATS  devel¬ 
opments  in  this  period.  Further  analysis  of  multiple  DoD  ATS  types  in  many  field  and  depot 
locations  shows  the  costs  to  develop  a  general  purpose  family  ATS  is  of  the  same  relative 
magnitude  as  the  cost  to  develop  ATS  for  a  single  unique  system  application. 


Figure  3  illustrates  R&D  cost  avoidance  opportunities  when  weapon  system  unique 
ATE  developments  are  avoided  by  requiring  the  use  of  a  general  purpose  standard  tester. 
Each  time  a  new  ATE  development  is  initiated,  the  cost  is  at  100%,  the  starting  point  of  the 


Figure  3.  Summary  of  ATE  R&D  Cost  Avoidance  Opportunities 
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curve.  The  total  R&D  cost  line  shows  that  R&D  costs  are  amortized  when  a  single  ATS  is 
used  in  multiple  applications:  the  current  R&D  amortization  point  for  ATE  such  as  CASS 
or  IFTE,  already  planned  for  application  to  do^ns  of  weapon  programs,  is  far  to  the  right 
off  of  this  chart  The  Total  R&D  line  includes  the  cost  for  additional  ATS  R&D  needed  to 
meet  additional  new  (unique)  weapon  system  testing  needs;  the  “New  Delta”  R&D  area  in 
this  figure  relates  to  the  average  5%  testing  capability  additions  that  are  expected  to  be 
required  for  subsequent  weapon  system  applications. 

Production  Unit  Cost  Reductions.  ATS  production  unit  cost  reductions  can  be 
achieved  by  larger  order  quantities.  Qassically,  production  unit  costs  are  quantity  depen¬ 
dent  due  to  two  principal  cost  categories:  recurring  and  non-recurring  costs.  The  recurring 
costs  are  direcdy  associated  with  the  time  and  materials  needed  to  actually  manufacture  an 
item.  Improvements  in  manufacturing  efficiencies,  which  result  firom  accumulated  experi¬ 
ence  as  additional  units  are  produced,  tend  to  reduce  the  recurring  cost  in  a  consistent  pat¬ 
tern  (production/manufacturing  learning  curve  theory).  Larger  and  more  stable  order 
quantities  for  materials  can  also  lead  to  cost  reductions.  The  non-recurring  costs  include 
fixed  expenditures  associated  with  doing  business  (i.e.,  sustaining  engineering,  facilities, 
overhead).  Pro-rating  these  fixed  costs  across  greater  quantities  results  in  reduced  produc¬ 
tion  unit  costs  due  to  economies  of  scale.  Consequently,  as  production  quantities  increase, 
the  opportuiuties  to  decrease  production  unit  cost  increase. 

Historical  data  for  the  15  weapon  systems  reviewed  shows  DoD  “unique  type  ATE” 
procurements  have  average  total  production  quantities  of  23  each.  By  contrast,  currently 
plaruied  production  quantities  for  IFTE  and  CASS  are  approximately  200  and  500  (through 
FY  99)  respectively.  ATS  manufacturers  of  both  IFTE  and  CASS  estimated  that  average 
unit  production  costs  will  decrease  by  30%  when  ATS  quantities  increase  from  50  to  500. 

Reduced  ATS  Production  Quantities.  General  purpose  DoD  ATS  standard  fami¬ 
lies  have  the  capabilities  to  meet  a  wide  range  of  test  applications.  This  allows  reduced  ATE 
quantities  by  replacing  multiple  ATS  types  at  each  maintenance  location  and  loading  the 
maintenance  worldoad  more  efficiently  on  the  testers.  Examples  of  the  quantity  reduction 
impact  of  using  general  purpose  standard  ATS  is  summarized  in  Table  4. 
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Table  4.  Potoitial  ATE  Quantity  Reductions 


SYSTEM 

ATE  QUANTITY 
REDUCTIONS 

F.16 

(ATS  STUDY  ANALYSIS) 

50% 

F/A-18 

(NAVY  ACTUALS) 

82% 

F-15 

(SEPARATE  AF/ARMY  ANALYSES) 

72% 

•  F*16.  As  a  part  of  the  ATS  study  analysis,  an  evaluation  of  the  F-16  testers  tech¬ 
nical  capability  and  woikloading  was  performed.  This  analysis  showed  that  use 
of  a  general  purpose  tester  CASS  to  accommodate  workloading  could  result  in 
reducing  tester  quantities  by  approximately  50%  over  the  current  quantity  of 
testers  required. 

•  F/A-18.  The  Navy  is  currently  replacing  6  different  F-18  ATS  with  the  CASS. 
The  improved  workloading  that  will  result  allows  the  F-18  to  reduce  from  356 
total  testers  to  64  CASS  testers,  a  quantity  reduction  of  82%.  The  356  number 
represents  F/A-18  testers  that  evolved  over  time  for  which  there  exist  some 
duplication  and  overlapping  ctq>abilities  that  the  Navy  did  not  eliminate. 

•  F-15.  Separate  analyses  (performed  by  both  the  Army  and  Air  Force  outside  of 
this  ATS  study)  have  shown  the  potential  to  replace  at  least  6  (and  potentially 
8)  of  the  current  F-15  testes  with  the  IFTE.  The  workload  realignment  associ¬ 
ated  wid)  the  6  to  1  replacement  would  lead  to  a  72%  reduction  in  tester  quan¬ 
tities  over  the  number  currently  required. 

2,2,2  Operation  and  Support  Benefits 

The  Services  do  not  collect  complete  operation  and  support  (O&S)  information  as 
it  relates  to  automatic  test  systems;  therefore,  die  analysis  and  conclusions  in  this  area  relied 
on  ATS-speci6c  anecdotal  information.  Although  a  rigorous  cost  analysis  was  not  possible 
given  the  available  data,  previous  studies  performed  by  die  Army  and  Navy  clearly  show 
that  ATS  support  is  more  effectively  provided  with  common  general  purpose  family  versus 
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iq[)plication  specific  (peculiar  or  unique)  ATS.  Examples  of  O&S  benefits  were  collected  for 
the  five  areas  presented  in  Table  S. 


Table  5.  Examples  ATS  Family  Support  Benefits 


O&S  Category 

Examples  of  Benefits 

Technical 

Publications 

CASS  will  reduce  the  total  technical  manual  requirements  for  5  cur¬ 
rent  automatic  test  systems  from  367  to  13.  Technical  manual  infor¬ 
mation  will  be  on  optical  disks  which  will  further  improve  efficiency 
over  traditional  paper  control  and  change  management. 

Support  of 
ATS 

Each  type  of  ATS  requires  a  unique  set  of  between  80-200  line  items 
of  support  equipment  CASS  requires  135  items  across  its  four  con¬ 
figurations,  Aus  reducing  a  comparable  number  of  line  items  for 
each  duplicative  ATS  type  eliminated. 

Manpower, 
Personnel  & 
Training 

CASS  estimated  personnel  savings  of  40%  and  training  cost  reduc¬ 
tions  of  75%. 

IFTE  estimated  personnel  savings  of  42%  and  training  cost  reduc¬ 
tions  of  52%. 

Supply 

Support 

Navy  projected  that  the  total  number  of  ATS  related  spare  parts 
could  reduced  from  30,0(X)  to  3,800  per  aircraft  carrier  by  replac¬ 
ing  automatic  test  systems  with  CASS. 

Facilities 

Navy  estimated  that  facility  space  could  be  reduced  by  approximate¬ 
ly  5,000  square  feet  on  a  typical  aircraft  carrier  by  replacing  existing 
automatic  test  systems  with  CASS. 

Although  the  exact  benefits  could  not  be  quantified,  the  authors  observed  that  com¬ 
mon  ATS  families  would  allow  flexibility  of  tester  work  loading,  and  possibly  reduce  total 
ATS  requirements.  Of  the  ATS  acquisition  strategy  options  evaluated,  die  common  ATS 
families  strategy  was  the  only  option  that  had  the  potential  to  increase  flexibility  of  weapon 
system  support,  reduce  deployment  logistics  tails,  and  permit  more  flexible  use  of  mainte¬ 
nance  personnel. 

23  ATS  FAMILy  CAPABILITIES 

Existing  ATS  standard families,  IFTE  &  CASS,  meetDoD  automatic  testing 
technical  requirements  today  and  will  handle  nearly  all  projected  testing  re¬ 
quirements  through  the  next  decade  with  only  modest  improvements. 

Test  requirements  analyses  of  existing  and  future  cotr^lex  weapon  systems  (F-15, 
F/A-18,  AH-64,  F-22,  and  A-12)  showed  that  IFTE  and  CASS  exceeded  95%  of  technical 
test  requirements.  CASS  test  stimulus  and  measurement  capabilities  at  the  front  panel 


interface  were  specified  (in  1986)  to  take  advantage  of  anticipated  1992  test  instrument 
capabilities.  Tliese  CASS  specification  requirements  represented  an  aggressive  design 
challenge  to  attain  anticipated  1992  test  instrument  stimulus  and  measurement  capabilities 
out  at  the  front  panel  interface  to  items  being  tested  concurrent  with  the  CASS  design  com¬ 
pletion.  All  of  the  instruments  developed  in  the  downsized  Modular  Measurement  System 
(MMS)  instrument  packaging  standard  for  the  CASS  were  designed  and  developed  during 
the  last  three  years.  The  currency  of  this  set  of  general  purpose  test  resources  cannot  be 
exceeded  in  fielded  equipment  today. 

The  IFTE  Base  Shop  Test  Facility  (BSTF)  is  a  general  purpose  ATS  and  in  addition 
meets  the  Army’s  high  mobility  requirements.  The  current  IFTE  autonutic  test  system  is 
size  (weight  and  volume)  competitive  with  paper  designs  coming  off  drawing  boards  today. 
The  proposed  DoD-designated  ATS  family  members  (IFTE  and  CASS)  together  represent 
the  forward,  leading  edge  of  testing  technology  capabilities  in  inventory  today  for  highly 
mobile,  intermediate  level,  depot,  and  factory  acceptance  testing. 

Analysis  of  DoD’s  most  modem  ATS  shows  the  CASS  tester  meets  the  broadest 
range  of  testing  cjq)abilities.  No  other  tester  evaluated  covered  the  depth  and  breadth.  IFTE 
was  a  close  second  and  adds  the  additional  dimension  of  a  highly  mobile  single  ATS  sys¬ 
tem. 

The  broad  capabilities  of  these  two  testers  are  illustrated  by  the  large  numbers  of 
items  to  be  tested  on  each.  Over  1250  units  under  test  (UUTs)  have  been  funded  or  project¬ 
ed  for  the  Navy  CASS  Introduction  Plan  (QP)  [AIR  552,  30  April  1993]  which  includes 
off-load  or  new  program  start-ups.  Furthermore,  nine  existing  ATS  station  types  are  being 
replaced  by  the  CASS  ATS.  The  Army  has  funded  or  projected  239  units  under  test  for  the 
near-term  (Fiscal  Years  (FY)93-95)  IFTE  BSTF  laydown  plan  [PM  IFTE].  The  IFTE  BSTF 
will  replace  two  existing  Army  ATS  types. 

2.3.1  IFTE  and  CASS  Technical  Capabilities 

IFTE  and  CASS  were  developed  as  general  purpose  automatic  test  systems  and 
designed  to  be  configured  into  broad  coverage  ATS  families.  The  Army,  Navy,  and  Marines 
performed  extensive  mission  and  test  requirements  studies  including  assessment  of  the 
capabilities  of  fielded  ATS  and  future  mission  and  test  requirements  across  many  weapon 
systems.  These  studies  resulted  in  a  new  test  capability  requirements  envelope  for  general 
purpose  ATS.  The  ATS  development  programs  for  the  IFTE  and  CASS  met  or  exceeded  the 
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advanced  test  requirements  envelope  defined  by  these  studies  [NAEC-MISC-82-0147,  PM 
IFTE  Test  Reqts.  Analysis]  as  well  as  mission  deployment  requirements. 

Tables  6  to  10  summarize  current  IFTE  and  CASS  capabilities  and  identify  the  type 
and  range  of  projected  testing  requirements  that  may  be  desired  over  the  next  decade.  These 
tables  address  how  to  meet  potential  requirements  and  what  the  ATS  impacts  might  be. 
Impacts  are  categorized  as  none,  minor,  modest,  and  major: 

•  No  new  requirements  over  this  period  or  the  requirements  will  not  impact  the 
automatic  test  systems  (none). 

•  A  minor  impact  is  defined  as  a  change  or  extension  of  test  operation  software 
and  the  addition  or  exchange  of  a  few  test  resources  that  will  not  affect  the  over¬ 
all  ATE  fooq)rint 

•  A  modest  impact  is  defined  as  the  addition  of  new  chassis  resources  which  may 
lead  to  extension  of  the  ATE  footprint  after  a  few  test  resources  are  implement¬ 
ed. 

•  A  major  impact  is  defined  as  a  new  configuration  with  an  enlarged  fooqjrint  of 
20%  or  more. 

The  ATS  family  capabilities  are  summarized  against  five  of  the  primary  ATE  ele¬ 
ments  identified  in  the  introduction:  interface,  stimulus,  measurement,  control  and  switch, 
and  test  operating  system.  One  modest,  no  major,  and  a  range  of  minor  impacts  were  iden¬ 
tified. 

2  Ability  to  Use  and  Extend  IFTE  and  CASS  Capabilities 

An  analysis  of  cross-weapon  and  cross-Service  ATS  requirements  demonstrated 
that  IFTE  and  CASS  exceed  most  technical  test  requirements.  More  importantly,  since 
IFTE  and  CASS  use  flexible  industry  standard  architectures  that  promote  modularity  and 
partitioning  (e.g.,  MMS,  VME,  Ethernet),  they  may  be  updated  relatively  easily.  As  signif¬ 
icant  performance  enhancing  technologies  become  available,  the  families  can  incorporate 
the  improvements  widiout  disturbing  basic  architectures.  Changes  to  IFTE  or  CASS  to 
achieve  new  testing  capabilities  may  be  accommodated  with  only  modest  impact.  For 
example,  the  Navy  is  funding  an  Air  Force  managed  joint  Service  initiative  to  add  full-up- 
round  radar  guided  missile  testing  capabilities  to  the  CASS  with  only  modest  changes. 

The  cross- weapon  and  cross-Service  review  compared  F-18  to  IFTE,  F-22  to 
CASS,  and  Apache  to  the  F-16  ATS.  The  Army  and  Navy  were  asked  to  select  assemblies 


that  represent  a  challenge  for  other  automatic  test  systems.  Since  actual  test  requirements 
for  an  F-22  system  have  not  been  determined  yet,  the  F-22  Conunon  Automatic  Test  Sys- 
tnn  (CATS)  preferred  equipment  list  was  used  as  a  comparison  baseline.  Three  F-18  weap¬ 
on  Fq)laceable  assemblies  (WRAs)  were  evaluated  against  the  IFTE  Base  Shop  Test 
Facility. 

Fiiuilly,  to  gain  insight  into  the  capability  of  existing  automatic  test  systems  to  sup¬ 
port  cross-Service  testing  requirements,  three  Apache  line  replaceable  units  (LRUs)  were 
evaluated  against  the  F- 16  Improved  Avionics  Intermediate  Station  (lAIS).  Comparisons 
results  are  presented  in  Thble  11.  The  “Exceptions”  in  this  table  are  grouped  as  none,  minor, 
modest,  and  major  (same  detinitions  used  in  Section  2.3.1  now  applying  to  exceptions). 
Few  minor  and  no  modest  nor  major  exceptions  were  identified. 

Additional  comparisons  of  DoD  tester  capabilities  are  presented  in  the  Appendix  D. 
ATE  Comparisons.  Based  on  the  analyses  from  these  comparisons,  only  minor  tester  capa¬ 
bility  technical  improvements  to  ATS  families  will  be  required  over  the  next  decade. 


Table  7.  Summary  of  DOD  ATS  Famiiy  Capabilities  of  Stimulus  Elements 


CAPABILITIES 

FUTURE 

REQUIREMENTS 

IMPACT 

• 

Arintnry  waveforms  to  200 
mbits/sec  (CASS),  SO  mbits/ 
scc(IFTE), 

None 

Digital  -  all  logic  families 

336 10  to  40  mbit  (CASS) 

224 10  to  SO  mbit  (IFTE). 

Digital  pins  approximtue  S12 
range,  data  rates  to  SO  mbit, 
clock  rates  to  100  MHz. 

Minor  -  Near-term  CASS  expandable  to 
432  I/O  at  40  mbit. 

Modest  -  IFTE  expandable  to  1,024  IA3 
at  SO  mbit.  1/2  rack. 

Minor  -  TPS  port^ility  must  be  man¬ 
aged. 

Minor  -  Develop  a  procuraUe  qrecifica- 
tion  for  a  next  generation  digital  test  unit 
(DTU)  replacement  subsystem  common 
for  all  DoD  designated  ATS. 

• 

• 

Analog  functions  -  0.01  •>  2S 
MHz. 

None 

• 

Communications  waveforms 
available  include  standard 
AM  (ILS.VOR),  TACAN. 
Linear  Pulse.  FSK,  MSK. 
BPSK,  QPSK,  OQPSK 
(GPS),  genok  spread  spec- 
tnim  hardware  (IFTE). 

Improve  programmability  of 
qxead  spectrum  resources 
and  ATLAS  integration. 

Minor  -  Upgrade  existing  test  resources 
to  achieve  higher  programmability. 

• 

Synthesizers  to  40  GHZ 
(CASS),  22  GHz  (IFTE) 
fast  switching  to  18.4  GHz 
(CASS),  2.3  GHz  (IFTE). 

Extended  synthesizer  range 
(Navy). 

Minor  •  Extended  range  synthesizers, 
will  be  integrated  as  COTS  items  rein¬ 
ing  existing  instrument  assets.  No  foot¬ 
print  change. 

• 

EO  collimator  0.S  ->  S.0  inch 
Wavelength  1.064  urn  and 
IR.  visual  targets  and  tracker 
stimulus. 

EO  Aperture  to  17  inch 
(Apache). 

Minor  -  Assess  &  develop  EO  aper¬ 
tures.  standard  EO  interface  to  simulate 
larger  aperture. 

• 

Extended  laser  firequencies  to 
far  IR  (Army). 

Minor  -  Procure  odter  laser  firequencies: 
hardware  &  software  already  supports 
laser  exchange  in  field  use. 

Advanced  target  generators 
(Missiles). 

Minor  -  Develop  advanced,  inogram- 
mable  target  generators, 
modify  existing  EO  tester  target  genera¬ 
tion. 

• 
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TaMe  8.  Summary  of  DOD  ATS  Family  Capabilities  of  Measurement  Elements 


CAPABIUTIES 

FUTURE 

REQUIREMENTS 

IMPACT 

Digital  (see  Stimulus  Ele¬ 
ments  in  Table  7;  all  capabil¬ 
ities  are  bidirectional). 

(See  Stimulus  Elements). 

Minor  -  (See  Stimulus  Elements). 

SynchroAesolver  360  d^ree, 
130Vrms. 

None. 

Multimeter  to  lOOOV 
(CASS).  200V  (IFTE). 

None. 

Frequency  to  0.0001Hz 
(IFTE).  0.001  Hz  (CASS),  to 
200  MHz  (CASS).  lOOMHz 
(DTE). 

None. 

Digitizer  to  SOOMHz 
(CASS).  50  MHz  (IFTE). 

None. 

Demodulators  for  standard 
AM  (ILS.VOR).  TACAN. 
Linear  Pulse.  FSK.  MSK. 
BPSK.  QPSK.  OQPSK 
(GPS),  generic  qnead  spec¬ 
trum  hardware. 

Improved  abstraction  level 
for  test  software  interface 
(lower  cost  TPS  r^iost  to 
designated  ATS). 

Minor  -  Coimnunications:  devd<^ 
maintain  DoD  common,  higher  level  test 
libraries.  This  will  lower  recurring  TPS 
engineering  cost 

RF  Power -70  to-t44  dbm  50 
GHz 

RF  VectOT  voltmeter  to  1 
GHz 

RF  Peak  power  to  26.5  GHz 
RF  Network  analysis  to  26.5 
GHz. 

Increased  precision  phase 
noise  measurement 

Minor  -  Phase  noise  measurement 
option  based  upon  COTS  instruments. 
possiUe  1/5  rack  addition. 

EO  collimator  0.5  ->  5.0  inch 
Wavelength  1.064  urn 
&  laser  detector.  RS  343. 
RS170.  Raw  Video  to 
40MHz. 

None. 

TaUe  9.  Sttimnary  of  DOD  ATS  Family  Capabilities  of  Control  and  Switch  Elemaits 


Table  10.  Summary  of  DOD  ATS  Family  Capabilities  of  Test  Equipment  Operating 

System  Elements 


CAPABILITIES 

FUTURE  REQUIRE¬ 
MENTS 

IMPACT 

ATLAS,  Non-ATLAS 

Add  ATLAS  1989/1994 

Minor  -  ATLAS  TPS  portability 

modules  in  Ada  or  For- 

capability  (Operational 

must  be  managed  through  con- 

• 

tran. 

Commonality). 

current  support  of  multiple  stan¬ 
dards,  software  maintenance  on 

ATLAS  Complex  RF 
waveform  extensions. 

designated  ATS. 

• 

Teradyne  IEEE 

ABBET  and  ATLAS 

Minor  -  Develop  ABBET  speci- 

WAVES  interface. 

compatibility  (to  lower 

fications  and  demonstrate  tool 

cost  of  TPS  develop- 

suites  on  designated  ATS. 

LASAR  V6  or  Teradyne 
binary  tile  format  for 

ment). 

• 

digital. 

Maintenance  data  inter¬ 
face. 

• 

Interactive  technical 
manual  interface  and 
functions. 

Built-in  test  concurrent 

Multiple,  concurrent  test 

Minor  -  Develop  and  demon- 

# 

and  on-demand,  auto- 

programs  execution. 

strate  on  designate  ATS  ATLAS 

matic  calibration. 

Advanced  built-in-test 
interfaces. 

extensions  for  concurrency  and 
built-in  test  interfaces. 

• 

Real  time  test  control 

Minor  -  Provide  real-time  test 

interface  (Missile  Test). 

control  software  with  missile 
ATS. 
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TaUe  11.  Results  of  ATS  Cross-System  Comparisons 


SYSTEMS 

ITEMS  COMPARED 

EXCEPTIONS 

Radar  Set  Receiver-Exciter. 

Nrnie  -Full  capability  exists. 

Radar  Target  Data  Processor. 

F-lSWRAs 
to  IFTE 

FLIR  Receiver  &  Optics  Stabi¬ 
lizer. 

Minor  -Stimulus  and  measurement 
requirements  within  capabilities. 

-  IR/EO  testing  needs  HW  fixturing 
augmentation. 

-  Optics  stabilizer  needs  HW  aug¬ 
mentation  for  vibration  test  stand 
and  motor-powered  mechanical 
interface. 

VXl  Instruments. 

None  -  Comparable  capabilities. 

F-22  CATS 
Preferred 
Equipment 
List 

to  CASS 

Radio  Frequency  Equipment 

None  •  Technical  capabilities  com¬ 
parable,  CATS  large  rack  &  stack 
equipment,  CASS  downsized  MMS 
based. 

EO  Equipment 

None  •  (Capabilities:  CATS  limited, 
CASS  bro^  based. 

PNVS  Electronic  Unit  Assem¬ 
bly. 

None  •  FuU  capability  exists. 

Apache 

LRUs 

to 

F-16  lAIS 

PNVS  Night  Sensor  Assembly. 

None  •  Stimulus  and  measurement 
requirements  exist,  holding  fixture 
augmentation  needed. 

Day  Sensor  Assembly. 

None  -  TPS  augmentation  of  a  trans- 
forma*  (on  qualified  parts  list-QPL) 
needed. 

TADS  Night  Sensor  Assembly. 

None  -  Temperature  probe,  voltage 
divider  and  variable  temperature  tar¬ 
get  source  required  for  TPS  interface 
adapter. 

Key  to  acronyms:  Common  Automatic  Tea  System  (CATS),  Consolidated  Automated  Support  System 
(CASS),  Electro-Optical  (EO),  Forward-Looking  Inbmed  Radar  (FLIR),  Integrated  Family  of  Test  Equip¬ 
ment  (IFTE),  Hardware  (HW),  Improved  Avionics  Intermediate  Shop  GAIS),  Infrared  (IR),  Line  Rqtlace- 
aWe  Unit  (LRU),  Modular  Measurement  System  (MMS),  Pilot  Night  Vision  System  (H4VS),  Target 
Acquisition  and  Designation  System  (TADS),  Test  Program  Set  (TPS),  VME  Extended  for  Instruments 
(VM),  Weiqwn  Rqtlaceable  Assembly  (WRA) 
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2.4  DOWNSIZED  TESTER  APPROACHES 


Appreciable  size  reductions  of  modern  general  purpose  ATS  are  achievable 

only  by  eliminating  test  system  cc^bilities  or  by  introducing  highly  tailored 

design  solutions. 

Two  critical  findings  are  central  to  this  conclusion:  (1)  the  term  “downsized  tester" 
addresses  an  amorphous  concept  with  little  meaning  outside  of  the  context  in  which  it  is 
used,  and  (2)  tester  downsizing  approaches  are  technology  limited  and  directly  define  tester 
application  limits  by  trading  off  capability  for  smaller  size.  The  following  paragraphs 
present  the  context  that  the  term  “downsized  tester”  is  used  within  DoD,  provide  details  of 
tester  downsizing  approaches,  and  summarize  the  consequences  of  tester  downsizing. 

Downsized  Tester.  There  exists  no  clear  or  consistent  definition  of  a  downsized 
tester.  The  term  “downsized"  is  not  technically  defined,  and  so  is  meaningless  if  taken  out 
of  context  Mthin  DoD,  the  term  downsized  tester  most  often  represents  an  ATS  with  one 
or  more  of  the  following  attributes:  reduced  volume,  reduced  weight  and  reduced  deploy¬ 
ment  load  relative  to  another  tester  or  group  of  testers  that  meet  some  predetermined  set  of 
testing  needs. 

For  example,  the  Air  Force  F-15E  Mobile  Electronic  Test  Set  (METS)  has  been 
refeired  to  as  a  downsized  tester.  This  tester  was  acquired  as  an  alternative  to  three  existing 
F-15  aircraft  ATE  that  deploy  with  the  aircraft:  Computer  Test  Station;  Conununication, 
Navigation  and  Identification  (CNI)  Test  Station;  and  Indicators  and  Control  (l&C)  Test 
Station.  The  F-ISE  METS  alternative,  although  it  has  less  functional  testing  capability  than 
the  three  individual  testers,  is  21,000  pounds  lighter,  4,000  cubic  feet  smaller,  and  requires 
7  fewer  pallets  to  transport  than  the  other  three  test  stations. 

From  another  perspective,  a  tester  is  “downsized”  if  it  reduces  the  total  deployment 
loads,  usually  aircraft  pallets,  needed  to  transport  full  system  testing  capabilities.  In  this 
context,  IFTE  could  be  used  to  further  reduce  the  fiill  ATS  deployment  burden  of  the  F-15. 
IFTE  is  a  highly  mobile  van  tester  that  requires  only  2.5  pallets.  )iWth  only  modest  modifi¬ 
cation,  IFTE  would  exceed  testing  capabilities  of  the  existing  METS,  and  at  the  same  time 
could  replace  other  F-15  tester  requirements  for  electronic  warfare  (EW). 

Tester  Downsizing  Approaches.  Given  the  current  state  of  test  technology,  a 
downsized  tester  is  nearly  always  synonymous  with  limited  range  testing  capabilities.  This 
reduced  tester  functionality  manifests  itself  from  two  perspectives:  (1)  the  physical  elimi¬ 
nation  of  testing  capabilities  that  would  have  been  present  in  a  more  general  purpose  tester, 
and  (2)  the  unique  design  of  special  purpose  testing  functions  with  limited  testing  perfor- 
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mance  capabilities.  These  may  be  applied  separately,  or  in  combination,  to  produce  a  tester 
diat  meets  testing  needs  of  only  a  limited  class  of  items.  The  apparent  shrinking  of  a  tester’s 
size  is  almost  always  achieved  by  a  combination  of  three  methods; 

•  Reduction  of  Tester  Functionality.  Reduction  of  functionality  achieves  the 
greatest  potential  for  physical  tester  size  reductions  by  eliminating  or  excluding 
ATE  elements.  However,  this  method  tends  to  inhibit  consolidation  of  multiple 
tester  requirements  and  tends  to  increase  total  weapon  system  tester-related 
deployment  burdens  (i.e.,  greater  weight  and  volume,  and  multiple  application 
specific  testers  versus  a  single  general  purpose  tester).  The  IFTE  and  CASS  pro¬ 
grams  both  avoided  this  method,  and  specifically  designed  general  purpose 
testers  that  address  a  broad  spectrum  of  functional  requirements. 

•  Packaging  UUT  unique  test  resources  in  active  IDs.  The  packaging  of  test 
resources  in  an  active  interface  adapter  reduces  a  tester’s  physical  size,  but  tends 
to  increase  total  tester  deployment  weight  and  volume  burdens.  The  size  differ¬ 
ence  for  ATS  element  resources,  whether  installed  in  the  tester  or  ID,  is  minor. 
However,  total  tester  deployment  burdens  increase  when  element  resources  are 
replicated  in  multiple  ID  types.  The  IFTE  and  CASS  tester  designs  incorporate 
a  broad  range  of  test  resource  capabilities  specifically  intended  to  minimize  tihds 
duplicative  packaging  concern. 

•  Application  of  Newer  Technology.  The  physical  size  of  a  tester  may  be 
reduced  by  q)plying  new  technology  ATS  elements  that  take  advantage  of  more 
densely  packaged  components  (i.e.,  advanced  microelectronics  devices, 
advanced  device  packaging  approaches,  and  new  reduced  form  factor  architec¬ 
tures).  IFTE  and  CASS  make  extensive  use  of  advanced,  state-of-the-art,  tech¬ 
nologies  to  meet  design  requirements.  For  example,  no  further  downsizing  of 
instrument  back-plane  interfaces  used  on  IFTE  and  CASS  are  on  the  horizon  in 
the  electronics  industry.  However,  this  downsizing  approach  may  have  negligi¬ 
ble  influence  on  ATS  size  reductions  for  the  immediate  future,  because  these 
new  instrument  back-plane  interfaces  have  become  the  predominately  preferred 
architecture  in  both  military  and  commercial  tester  applications.  No  other  near 
term  tester  size  reductions,  driven  solely  by  the  application  of  newer  technology 
were  identified  during  this  study. 

Consequences  of  Downsizing.  The  consequences  of  downsizing  ATS  are  less  gen¬ 
eral  purpose  functionality,  reduced  system  performance,  higher  costs  (development,  pro- 
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duction,  and  support)  per  functionality,  and  greater  obsolescence  potential.  Figure  4 
presents  a  summary  of  design  trade-offs  that  influence  the  size  of  automatic  test  systems. 
The  design  alternatives  for  reducing  ATS  size  have  a  tendency  towards  design  solutions 
indicated  by  the  anows  in  this  figure.  The  design  trade-off  examples  are  discussed  in  the 
following  paragraphs. 


ATS  Design  IVade-ofT 

Consequences 

Application 

General  Dcalpi 

Purpose  ^  «pacl«c 

Less  functionality 

Limited  uses 

Greater  onset  of 
obsolescence 

Harder  to  add 
capabilities 

Architecture 

Flexible  . ""1^ 

Instruments 

iimicr  onsei  oi 
obsolescence 

Higher  devei,  prod, 

&  support  costs 

ITAs 

&IDs 

Passive  Active 

Higher  Ibst  Program 
Set  costs 

Higher  devei,  prod, 

&  support  costs 

Power 
Supplies  & 
Switches 

Large  liadlid 

Range  Range 

Less  ftinctionality 

Limited  uses 

[Technology  Umited] 

Computers 

Comparable  Opportunities 

N/A 

Figure  4.  Design  Alternatives  for  Reducing  ATS  Size 


•  General  Purpose  versus  Design  Specific.  The  physical  size  of  the  F- 1 8  Inter¬ 
mediate  Avionics  Test  Set  (lATS)  was  downsized  by  using  UUT-specific  stim¬ 
ulus  and  measurement  functions  in  place  of  general  purpose  instruments.  This 
approach  limited  fault  detection  and  fault  isolation  capabilities.  Furthermore, 


this  approach  liiniuxl  upgrade  C2q>ability  to  accommodate  changes  resulting 
from  evolving  avionics  systems.  The  F-18  lATS  applied  testing  process  used 
fault  trees  and  nominal  input/ou^ut  stimuli  that  sim  dated  functional  applica¬ 
tions  in  order  to  identify  sets  of  probable  faults.  This  resulted  in  ambiguity 
groups  of  potential  failed  items  that  were  further  isolated  manually  by  switching 
and  installing  maintenance  assist  modules  (MAMs)  that  represented  known 
“healthy”  examples  of  the  suspect  units  under  test.  The  difference  in  applied 
functionality  is  further  illustrated  by  comparing  the  F-15E  Mobile  Electronic 
Test  Set  (METS)  downsized  automatic  test  system  with  the  Navy  CASS.  The 
difference  in  applied  functionality  is  several  orders  of  magnitude:  the  F-ISE 
METS  tests  22  line  replaceable  units  (LRU)  relative  to  current  estimates  of  over 
1,250  subsystems  and  assemblies  to  be  tested  by  CASS. 

•  Flexible  versus  Rigid  Architecture.  New,  more  rigid  architectures  will  be 
required  if  smaller  form  factors  than  may  be  accommodated  by  either  VXI  or 
MMS  are  desired.  VXI  stands  for  VME-bus  Extension  for  Instrunnents,  and 
MMS  stands  for  Modular  Measurement  System.  These  smaller  form  factors 
require  unique  design  solutions  with  tailored  architectures  that  use  non-standard 
protocols  and  non-standard  mechanical  and  electrical  interfaces.  Tailored  archi¬ 
tectures  tend  to  restrict  the  flexibility  of  the  ATS  to  acconunodate  new  testing 
functions  since  the  number  of  available  instruments,  especially  CXDTS,  is  sub¬ 
stantially  reduced.  Consequently,  there  is  a  greater  potential  for  obsolescence 
due  to  diminishing  manufacturing  sources  as  production  quantities  decline.  Fur¬ 
thermore,  unique,  non-standard  architectures  tend  to  complicate  design  and 
modification  processes. 

VXI  &  MMS  buses  are  examples  of  the  most  recent  developments  of  flexible 
instrument  interface  architectures.  This  level  of  technology  is  found  in  both 
IFTE  and  CASS.  At  present  these  two  instrument  interface  standards  represent 
the  smallest  general  purpose  form  factor  available.  Their  flexibility  is  derived 
principally  by  the  wide-range  of  commercial  off  the  shelf  (COTS)  instruments 
available  and  compatible  with  these  architectures.  No  further  downsizing  of 
these  or  other  instrument  back-plane  interface  architecture  standards  are  on  the 
horizon.  According  to  key  DoD  suppliers,  the  industrial  base  has  not  recovered 
the  R&D  that  went  into  these  downsizing  improvements,  and  therefore,  devel¬ 
opment  resources  to  support  further  downsizing  are  unlikely.  Furthermore,  they 
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perceive  that  the  need  does  not  exist  to  further  reduce  the  integrated  ATS  form 
factor  at  this  time. 

COTS  versus  Unique-Developed  In^ruments.  Instruments  that  meet  rigid 
architectures  or  special  limited  (non-general  purpose)  stimulus,  measurement, 
power,  or  load  ranges  are  generally  not  available  as  COTS  products.  Non-stan¬ 
dard  and  uncommon  specifications  require  unique  developments  with  commen¬ 
surate  higher  development,  production  and  support  costs.  Uniquely  specified 
products  tend  to  be  more  expensive  because  non-recurring  costs  are  not  amor¬ 
tized  across  multiple  2q)plications.  The  causes  for  a  greater  rate  of  obsolescence 
both  for  the  individual  instruments  and  the  integrated  ATS  are  the  same  as  those 
described  in  the  prior  paragraph. 

The  dominant  test  equipment  technical  and  engineering  capability  resides  with 
the  commercial  tester  and  instrument  manufacturers.  As  an  example,  instrument 
vendors  have  delivered  (X)TS  resources  that  were  developed  within  the  last 
three  years  to  CASS.  Tlie  vendors  obrerved  that  the  currency  of  the  general  pur¬ 
pose  test  instrument  resources  packaged  in  MMS  for  CASS  represents  the  per¬ 
formance  limits  available  eidier  as  COTS  instruments  or  as  uniquely  developed 
instruments.  In  general,  CX)TS  instruments  meet  or  exceed  the  technical  perfor¬ 
mance  requirements  and  possess  less  cost  and  programmatic  risks  than  present¬ 
ed  by  the  development  of  specialized  ATS  instruments. 

Active  versus  Passive  ITAs  &  IDs.  Smaller  (downsized)  ATS  form  factors 
have  often  been  achieved  by  placing  functional  capabilities  of  principal  ATE 
elements  in  active  interface  test  adapters  (ITA)  or  interface  devices  (ID).  For 
example,  additional  functional  capabilities,  such  as  an  extra  RF  interface,  might 
be  needed  at  a  unit  to  be  rested.  Consequently,  the  ID  must  incorporate  the  cor¬ 
responding  instruments  needed  for  each  functional  capability  not  provided  by 
the  ATE. 

A  general  purpose  ATE  design  does  not  preclude  requirements  for  active  IDs  to 
meet  each  and  every  test  requirement  However,  the  tendency  is  for  a  general 
purpose  tester  to  meet  more  of  required  resting  functionality  than  a  downsized 
tester,  since  greater  range  and  depth  of  the  functional  testing  capabilities  are 
incorporated  as  elements  of  the  general  purpose  ATE. 


Data  collected  shows  that  test  program  sets  are  more  expensive  for  uniquely  tai¬ 
lored  downsized  testers  with  active  interface  test  adapters.  The  cost  for  IPS 
development  and  FTA  production  was  found  to  be  approximately  two  times  as  # 

high  for  downsized  testers  than  for  full  sized  general  purpose  automatic  test  sys¬ 
tems  (e.g.,  F-18  lATS  vs.  F-18  MV). 


•  Large  Range  versus  Limited  Ran^  Power  Supplies  and  Switches.  IFTE  and 
CASS  incorporate  a  large  range  of  UUT  programmable  power  supplies  and 
loads.  Both  were  designed  to  meet  a  wide  range  of  power  and  switching  require¬ 
ments  that  are  found  across  multiple  weapon  systems.  In  general,  as  the  range 
of  power  supply  and  switching  features  shrink,  there  is  a  corre^onding  limit  to 
the  testing  applications. 


CASS  incorporates  power  conditioning  features  that  regulate  and  control  input 
power.  Also,  CASS  has  backup  powa-  features  that  ensure  if  power  is  lost,  the 
station  will  orderly  shut  down  and  save  data  under  controlled  conditions.  These 
power  features  makeup  over  20%  of  the  core  CASS  station.  Power-related  fea¬ 
tures  could  be  removed,  provided  by  facility  sources,  or  tailored  to  meet  UUT 
specific  testing  needs.  However,  if  removed,  CASS  would  lose  some  of  its  cur¬ 
rent  testing  capabilities.  Similarly,  the  range  of  ATS  stimulus  and  measurement 
capabilities  may  be  limited  by  selectively  deleting  building  block  components, 
instruments,  or  other  principal  ATS  elements.  In  the  case  of  the  DFTE,  the 
weight  and  volume  of  the  generators  (or  an  equivalent  power  source)  must  be 
accounted  for  in  the  total  IFTE  form  factor  for  opo’ational  applications. 


These  design  trade-offs  are  illustrated  by  the  following.  Recently  the  Marine  Corps 
outlined  requirements  for  a  downsized  automatic  test  system.  Their  initial  perception  was 
that  the  form  factors  of  IFTE  and  CASS  were  excessive  relative  to  operational  require¬ 
ments.  However,  upon  in-depth  review  of  testing  requirements,  all  of  the  CASS  general 
purpose  capabilities  were  required.  Furthermore,  packaging  all  of  these  c^abilities  in  mul¬ 
tiple  ATE  would  have  would  have  duplicated  common  ATE  elements,  and  resulted  in  a 
larger  total  ATE  volume.  Therefore,  the  Marines  decided  to  install  the  CASS  stations  in 
vans  for  mobile  use.  Since  CASS  was  to  be  repackaged,  the  Marines  discussed  the  feasibil¬ 
ity  of  deleting  some  levels  of  testing  functionality  in  order  to  achieve  a  reduced  form  factor. 
The  Marine  Corps  decided  they  needed  full  specified  functionality  to  meet  mission  require¬ 
ments.  Therefore,  a  single  composite  CASS  station  occup3dng  6  racks,  plus  another  half 
rack  for  EO,  will  be  used  to  meet  their  mobile  ATS  requirements.  This  provides  an  example 
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of  the  flexible  architecture  benefits  to  acconunodate  change,  yet  permit  the  retention  of  a 
common  ATS  standard  family. 

Further  downsizing  of  any  full  range  general  purpose  tester,  including  the  IFTE  and 
CASS  automatic  test  systems,  without  reducing  testing  functionality,  would  require  sub¬ 
stantial  technology  Ineakthroughs  in  areas  such  as  power  supplies,  and  switches.  Converse¬ 
ly,  as  requirements  to  add  testing  functionality  to  a  given  downsized  ATS  grow  broader, 
either  flie  physical  size  of  the  tester  begins  to  approximate  a  full  function  (e.g.,  general  pur¬ 
pose)  tester  or  the  combined  volume  of  the  tester  and  the  required  additional  active  inter¬ 
face  test  adapters  approximates  or  exceeds  fliat  of  a  full  function  tester. 

2,5  IMPACT  OF  WEAPON  SYSTEM  BUILMN-TEST 

Built-in-Test  (BIT)  will  not  substantially  change  ATS  requirements  for  off- 

equipment  repair  over  the  next  decade. 

BuUt-in-Test  will  not  eliminate  off-equipment  testing  needs.  Built-in-Test  provides 
a  valuable  function  as  one  of  several  tools  for  detecting  and  analyzing  system  faults;  how¬ 
ever,  it  does  not  have  the  same  accuracy  and  range  of  testing  capabilities  as  modem  general 
purpose  automatic  test  systems  when  used  for  off-equipment  testing. 

The  capability  of  Built-in-Test  to  identic  and  isolate  problems  is  limited  and  highly 
system  dependent  Where  BIT  technology  is  most  mature,  most  notably  digital  applica¬ 
tions,  there  are  still  significant  fault  coverage  limitations.  These  limitations  generally  fall 
in  two  categories:  (1)  performance  verification  over  the  broad  range  of  environment  and 
signal  timing  excursions  is  not  practical  with  current  digital  BIT  technologies,  and  (2)  the 
standards  needed  are  not  widely  nor  universally  applied. 

The  first  category  is  due  to  size  and  complexity  limitations.  Implementing  capabil¬ 
ities  equivalent  to  a  digital  test  unit  (DTU)  of  «milar  complexity  and  performance  as  found 
on  the  CASS  at  multiple  BIT  nodes  witiun  a  system  is  neither  practical  nor  feasible.  There¬ 
fore,  BIT  applications  are  limited  to  assessing  some  level  of  performance  around  a  nominal 
threshold  while  operating  under  normal  conditions. 

The  second  category  is  due  to  the  lack  of  widely  applied  standards.  Effective  digital 
Built-in-Test  requires  architectures  that  permit  dedicated  fault  detection  and  fault  isolation 
interface  and  commuiucation  with  chips,  multi-chip  carriers,  modules,  circuit  boards,  inter¬ 
face  busses,  and  system  protocols.  Review  of  the  F-22  aircraft  advanced  avionics  revealed 
that  close  to  a  quarto:  of  its  nucrociicuit  devices  may  be  commercial  or  nulitarized  com¬ 
mercial  products  (i.e.,  meet  extended  temperature,  environmental,  and  reliability  con- 
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straints)  that  do  not  incorpo^  BIT  int^ace  features.  Consequently,  major  portions  of 
critical  F-22  avionics  circuitry  must  either  be  void  of  BIT  or  must  be  redesigned  to  incor¬ 
porate  BIT  ciq)abiUties  such  as  boundary  scan  and  diagnostic  bus  interfaces. 

BIT  capabilities  that  fall  outside  the  digital  domain  pre»;nt  an  even  greater  chal¬ 
lenge.  The  ability  of  current  BIT  technology  to  detect  and  diagnose  to  a  unique  faulty 
replaceable  unit  or  component  rapidly  degrades  as  the  testing  spectrum  tu-oadens  to  mixed 
signal,  analog,  radio  frequency,  and  electro-optics  (EO).  For  example,  requirements  for 
high  radio  frequency  (RF),  and  microwave  exceed  most  current  BIT  technical  capabilities. 
Yet  future  weapon  systems,  as  illustrated  by  technology  distribution  projections  for  new 
aircraft  avionics  in  Table  12,  will  have  a  majority  of  subsystems  that  fall  in  die  domains 
where  Built-in-Test  is  either  limited  or  impossible  at  this  time. 


Table  12.  Technology  Distribution  for  Advanced  Avionics  Subsystems 


Aircraft 

Digital 

Analog  and 
Mixed  Signal 

RF  and  EO 

F-22 

24% 

43% 

33% 

A- 12 

42% 

33% 

15% 

Built-in-Test  is  advocated  as  the  dominate  approach  for  future  fault  detection  and 
fault  isolation  of  advanced  ground  electronic  and  avionic  systems.  The  principal  target  of 
BIT  designs  is  on-equipment  detection  of  faulty  removable  units.  While  this  may  lower  the 
number  of  Cannot  Duplicates  (CND)  seen  between  organizational-level,  intermediate-lev¬ 
el,  or  depot  maintenance,  it  does  not  address  die  testing  requirements  for  off-equipment 
automatic  test  systems.  In  fact,  proper  diagnosis  and  repair  of  systems  incorporating  Built- 
in-Test  requires  increased  automatic  test  system  complexity  vrith  special  test  features  and 
buses  for  built-in-test  hardware. 

Even  where  technicaUy  possible,  extensive  BIT  implementation  may  not  be  feasi¬ 
ble.  Built-in-Test  is  difficult  to  incorporate  because  it  requires  custom  engineering  for  each 
item.  Furthermore,  it  increases  costs,  reduces  reliability,  and  adds  additional  weight  and 
power  use.  Built-in-Test  is  subject  to  the  same  system  engineering  trade-off  analysis  as  any 
other  system  function  in  a  complex  system.  With  ever  improving  reliability,  on-equipment 
Built-in-Test,  other  than  for  performance  assessment,  will  often  be  deleted  in  an  overall 
system  performance  or  economic  trade-off  analysis. 
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OfT-equipmem  work  load  is  anticipated  to  decrease  because  of  increased  item  reli¬ 
ability.  Continuing  design  complexity  growth  is  resulting  in  decreasing  fault  isolating  capa¬ 
bility  of  Built-in-Test  for  off-equipment  testing.  This  condition  is  increasing  the 
engineering  complexity  of  off-equipment  test  program  set  development  atKl  ATS. 

2.6  TEST  SOFTWARE  DEVELOPMENT  &  SUPPORT  ENVIRONMENT 

Cost-effective  test  software  engineering  and  rehosting  requires  new  stan¬ 
dards,  and  development  and  support  environments  for  test  iifformation  and 
software. 

Software  development,  acquisition,  and  rehosting  represent  major  costs  for  ATS 
systems.  Although  hardware  still  represents  the  dominant  R&D  and  production  ATS  cost 
category  for  field  or  intermediate-level  maintenance  applications,  newer  systems  are  show¬ 
ing  a  trend  towards  increasing  software  percentages.  At  depots,  the  test  program  software 
costs  as  a  percentage  of  total  ATS  acquisition  costs  were  generally  higher  because  of  the 
favorable  utilization  that  depots  can  achieve  by  concentration  of  test  resources.  The  depot 
test  software  costs  often  averaged  50%  of  the  total  acquisition  costs.  Selected  examples  are 
summarized  in  Table  13. 

Table  13.  Software  Costs  as  a  Percentage  of  Total  ATS  Acquisition  Costs  for  Selected 


Applications 


ATS 

APPLICATION 

SOFTWARE  COSTS 
(Percentage  of  Total  ATS  Acquisition) 

F-18 

20% 

F-22 

32%  to  35% 

Depots 

-50% 

Costs  for  software  become  important  in  the  ATS  life  cycle  beyond  acquisition.  As 
an  ATS  ages,  it  often  must  be  replaced  with  more  modem  equipment  because  of  hardware 
obsolescence  and  non-availability,  or  new  capability  requirements.  The  cost  of  rehosting 
software  often  becomes  the  deciding  factor.  At  present,  the  Navy  is  investing  over  $800 
million  in  rehosting  software  from  six  aviation  testers  onto  the  standard  Navy  tester,  CASS. 
Although  this  price  tag  is  high,  it  represents  an  overall  cost  avoidance  relative  to  what  the 
total  costs  would  have  been  if  the  Navy  had  continued  to  modify  and  support  the  six  old 
testers  rather  than  off-loading  to  CASS. 


39 


Analysis  of  TPS  development  and  support  expense  categories  were  conducted,  and 
against  each  of  these  categories,  potential  cost  reductions  for  the  software  environment 
were  estimated.  The  estimates  were  formulated  around  potential  technology  improvements 
and  software  environment  standards  yet  to  be  developed.  Therefore,  the  cost  reduction  esti¬ 
mates  were  divided  in  to  near-  and  long-term  opportunities.  Table  14  summarizes  these 
results.  If  a  mature,  long-term  capability  was  available  and  applied  against  the  projected 
TPS  rehosting  burden  from  obsolete  automatic  test  systems  to  CASS  as  planned  by  the 
Navy,  the  cost  savings  would  exceed  $500  million. 

Table  14.  Test  Software  Development  and  Support  Environment  Cost  Benefits 


TIMEFRAME 

POTENTIAL  TPS 
DEVELOPMENT 
COST  REDUCTIONS 

Near  Term 
(5  - 10  years) 

17% 

Long  Term 
(10  -  20  years) 

70% 

Case  studies  revealed  that  the  development  and  implementation  of  common  ATS 
software  development  and  support  environment  would  reduce  total  weapon  system  ATS 
acquisition  costs  by  8%  to  10%.  (Note  that  this  represents  additional  acquisition  savings 
over  and  above  potential  savings  that  are  due  to  an  ATS  family  acquisition  strategy.) 

Without  the  development  of  new  common  standards  for  software  development  and 
support  environments,  the  only  currently  available  technical  solution  to  achieve  operational 
commonality  across  ATS  applications  is  for  each  ATS  to  have  100%  identical  assets. 

2.7  IMPACT  ON  INDUSTRY 

Applying  a  standard  ATS  families  acquisition  strategy  will  not  adversely  af¬ 
fect  DoD’s  ability  to  acquire  test  resources  from  industry. 

DoD  ATS  research  and  development  and  production  expenditures  are  spread  across 
a  large  number  of  contractors,  with  a  small  fraction,  typically  less  than  10%,  of  the  ATS 
market  in  any  one  firm.  Critical  automatic  test  technical  and  engineering  capability  resides 
in  the  commercial  sector  instrument  manufacturers,  and  more  recently  for  critical  automat¬ 
ic  test  technical  and  engineering  capability  for  test  subsystems  resides  with  commercial 
sector  ATS  providers.  In  no  case  do  DoD  sales  exceed  50%  of  the  sales  of  major  commer¬ 
cial  ATS  and  instrument  providers.  Furthermore,  many  of  the  instruments  and  ATS  sub- 
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systems  required  for  DoD  automatic  test  equipment  are  widely  required  in  the 
telecommunications  and  computer  industries. 

Only  a  few  companies  have  automatic  test  systems  as  their  principal  business,  and 
their  technical  capability  is  in  the  systems  integration  business.  The  most  recent  ATS  devel¬ 
opment  programs  are  relying  heavily  on  commercial  instruments  and  subsystems,  and  the 
value  added  has  been  in  the  design  integration,  system  packaging,  and  software  environ¬ 
ment. 

The  dominant  software  standard,  ATLAS,  has  been  co-developed  with  the  commer¬ 
cial  airline  industry.  There  are  numerous  test  software  environment  development  compa¬ 
nies,  many  of  which  are  snudl  businesses.  These  conditions  are  summarized  in  Table  15. 


Table  15.  Availability  of  ATS  Component  Sources 


ATS  COMPONENT 

SOURCE 

Test  instruments 

Multiple  COTS 

Sources 

TPS  software 

Broad  set  of  competitive  sources 

Test  operating  software 

A  few  emerging  commercial  suppliers 

Test  development  environment 

Commercial  CAE  companies  and  a  few 
emerging  integrated  test  environment 
commercial  suppliers 

ATS 

Prime  weapon  or  subsystem  providers 

Looking  at  total  acquisition  quantities  for  DoD  versus  the  conunercial  sector  and 
across  the  broad  sets  of  ATS  element  sources,  the  authors  concluded  that  a  DoD  ATS  family 
strategy  will  not  adversely  impact  ATS  industries.  Although  the  actual  amounts  are  propri¬ 
etary,  and  therefore  not  presented  here,  the  test  equipment  and  instrument  vendors  inter¬ 
viewed  observed  that  DoD  constitutes  only  a  small  percentage  of  their  market.  For 
example,  the  world  market  test  instrument  sales  are  over  $6  billion  annually.  The  delta  to 
these  world  wide  test  instrument  sales  attributed  to  the  DoD  ATS  estimated  savings  would 
average  less  than  1  %  per  year. 

From  another  perspective,  the  combined  1992  sales  of  VXI  and  MMS  instruments 
represent  less  than  2%  of  the  total  instrument  market.  If  adjusted  for  comparable  product 
market  share,  this  percentage  increases  to  approximately  16%.  Assuming  test  instruments 
and  other  test  resources  represent  about  50%  of  the  anticipated  savings  identified  by  the 
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analyses  in  this  study,  the  potential  impact  of  a  DoD  ATS  family  investment  strategy  will 
be  less  than  8%  of  this  specific  market 

TPS  software  is  supplied  by  a  broad  set  of  contractors  in  highly  competitive  pro¬ 
curements.  Consolidation  on  common  automatic  test  equipment  would  improve  TPS  qual¬ 
ity  by  providing  a  smaller  set  of  target  ATE  systems  to  focus  TPS  development  process  and 
third-party  tool  development  ATS  common  operating  software  and  TPS  development  soft¬ 
ware  are  developed  by  a  broad  set  of  computer-aided  engineering  (CAE)  vendors  to  both 
conunercial  and  defense  contractors. 

The  authors  observed  that  ATS  expenditure  reductions  of  the  relative  magnitude 
identified  in  this  study  might  be  attributable  solely  to  DoD  budget  reductions.  An  ATS  fam¬ 
ily  investment  strategy  was  found  to  provide  the  means  to  (1)  compensate  for  potential  bud¬ 
get  resource  reductions  and  (2)  meet  weapon  system  ATS  needs,  both  without  adversely 
affecting  operational  ATS  capabilities  nor  adversely  affecting  ATS  industry. 

2.8  PURPOSEFUL  REUSE  OF  ATS  RESOURCES 

Standard  ATS  families  will  promote  the  purposeful  reuse  of  test  equipment 

and  test  software  across  factory,  depot,  and  operational  maintenance. 

Standard  general  purpose  ATS  were  found  to  facilitate  and  promote  the  reuse  of 
ATE  and  TPS  elements.  Also  when  used  at  multiple  maintenance  levels,  common  ATS  fam¬ 
ilies  tend  to  improve  the  relative  diagnostic  accuracy  of  testing  results. 

The  opportunities  to  purposefully  reuse  test  equipment  and  test  software  are  many 
and  varied.  Automatic  test  systems  for  factory  in-process  and  acceptance  testing  are  typi¬ 
cally  developed  and  controlled  by  the  contractor.  The  authors  observed  that  the  entire  area 
of  factory  automatic  test  systems  might  prove  to  be  a  fruitful  area  for  future  investigation 
because  of  the  large  dollar  amounts  involved.  For  example,  AMRAAM  acquired  350  test 
stations  of  190  types  for  24  factory-level  locations,  mostly  special  purpose  with  unique 
software.  The  Joint  Air  Force  and  Navy  Advance  Medium  Range  Air-to-Air  Missile 
(AMRAAM)  program  reported  approximately  $350  million  for  special  test  equipment 
(STE),  of  which  around  $70  million  was  Government  furnished  equipment  (GFE)  special 
test  equipment  DoD  currently  has  listed  GFE  special  test  equipment  valued  at  $10.5  bil¬ 
lion.  Due  to  the  limited  availability  of  factory  ATS  data,  conclusive  observations  on  the 
benefits  of  common  in-process  factory  automatic  test  systems  were  not  possible.  To  be  con¬ 
sistent  with  other  categories  of  collected  and  sunvnarized  data,  this  factory  information  was 
not  included  with  the  weapon  systems  ATS  summary  data. 
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Early  selection  and  fielding  of  an  ATS  standard  family  will  help  to  avoid  costly 
interim  support  over  the  lifecycle  of  the  weapon  system.  Early  selection  (i.e.,  by  the  start 
of  the  EMD  phase)  ensures  that  an  automatic  test  systems  will  be  available  for  factory 
acceptance  testing  and  permits  sound  concutrent  engineering  practices  to  be  followed  in  the 
design  and  in^lementation  of  system,  diagnostic,  and  support  capabilities  elements.  Fre¬ 
quently  the  same  testing  capabilities  and  requirements  that  are  needed  for  factory  accep¬ 
tance  are  also  required  in  Service  depot  and  intermediate  maintenance  level  testing. 

ATS  for  intermediate  and  field  testing  requirements  have  historically  been  specified 
and  developed  late  in  the  EMD  or  production  phase.  Consequently  automatic  test  systems 
(and  especially  the  test  program  sets)  are  delivered  late,  relative  to  operational  need.  When 
this  occurs,  interim  contractor  support  is  used  to  compensate  for  the  lack  of  testing  capabil¬ 
ity.  If  an  ATS  family  is  designated  early,  the  tester  may  be  used  for  factory  acceptance, 
thereby  permitting  early  maturity  of  oiganic  depot  and  field  ATS  maintenance  capabilities. 
A  Center  for  Naval  Analysis  (CNA)  case  study,  analyzing  F/A-18  aircraft  test  and  support 
requirements,  estimated  an  interim  support  cost  avoidance  of  around  $900  million  would 
have  been  possible  had  the  required  automatic  tost  systems  been  deployed  concurrent  with 
the  weapon  system.  The  Navy  stated  that  concurrent  ATS  deployment  with  a  weapon  sys¬ 
tem  is  now  a  standard  goal  and  expects  to  use  this  approach  extensively  wifir  CASS. 

The  purposeful  reuse  of  tost  equipment  and  test  software  across  factory,  dq)ot,  and 
operational  maintenance  is  tied  directly  to  three  ATS  design  factors:  Test  Strategy,  Data 
Transportability,  and  ATS  Flexibility.  Using  the  designated  ATS  families  at  multiple  appli¬ 
cation  levels  reduces  potential  limitations  for  reuse  of  TPS.  In  addition,  the  greater  the  flex¬ 
ibility  of  the  ATS,  the  greater  the  potential  of  reusing  tost  equipment  and  tost  software 
across  factory,  depot,  and  operational  maintenance. 

ATS  flexibility  describes  an  ability  of  the  testing  system  to  adapt  to  different  UUT 
tost  requirements.  In  general,  it  is  easier  to  transport  TPS  elements  firom  one  automatic  tost 
system  to  another  of  equal  or  greater  flexibility  provided  they  both  have,  or  may  incorpo¬ 
rate,  capabilities  to  cover  the  same  testing  requirements  spectrum.  The  effect  of  ATS  flex¬ 
ibility  on  the  ease  of  transporting  TPS  elements  may  be  addressed  from  several 
perspectives:  the  ability  of  the  automatic  tost  systems  to  accommodate  new  requirements 
at  flexible  physical  interfaces,  the  ability  to  accommodate  the  same  range  of  stimulus  and 
measurement  performance  properties  across  ATE  versions,  and  the  degree  to  which  a  TPS 
specification  language  will  accommodate  common  syntax  and  common  testing  methods. 
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Reuse  of  TPS  elements  is  severely  impeded  when  target  automatic  test  systems  are  neither 
identical  nor  incorporate  features  that  foster  ATS  flexibility. 

The  standard  ATS  approach  avoids  the  need  to  account  for  differences  because  the 
equipment  is  identical  at  all  levels.  Since  it  is  unlikely  that  total  standardization  could  (or 
should)  be  achieved,  highly  flexible  architectures  that  acconmtodate  a  wide  range  of  gen¬ 
eral  purpose  testing  needs  are  most  desirable.  The  advantages  of  standard  ATS  families  and 
flexible  general  purpose  tester  architectures  are  both  available  with  CASS  and  IFTE. 
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3.  ANALYSES  OVERVIEW 


Analyses  were  conducted  in  conjunction  with  the  DoD  Automatic  Test  System 
Investment  Strategy  Study.  Reviews  of  ATS  costs,  use,  policies,  processes,  and  organiza¬ 
tions  were  conducted,  involving  over  100  hours  of  technical  interchange  meetings  (TIMs) 
and  OSD/Service  Executive  Steering  Group  (ESG)  meetings  between  February  and 
November  1992.  Data  were  collected  and  analyses  performed  by  a  study  team  of  12  full¬ 
time  team  members,  with  siq)port  from  more  than  80  representatives  of  DoD  and  industry. 
IDA  continued  collecting  data  and  performing  analyses  through  March  1993. 

The  study  was  partitioned  and  oiganized  as  sevaal  separate  analyses.  The  follow¬ 
ing  provides  an  overview  of  each  analysis  and  highlights  their  interrelationships. 

Section  4,  Baseline  Data.  ATS  data  from  IS  wetyron  systems  were  collected  and 
subjected  to  detailed  analysis.  Principal  results  included  summaries  of  ATS  costs  and  quan¬ 
tities,  percentages  of  ATE  and  TPS  development  and  production  costs,  and  ratios  of  ATS  to 
wetqron  system  production  costs  by  system  type.  Results  were  used  to  establish  a  baseline 
for  subsequent  analyses. 

Section  5,  Ahemative  ATS  Investment  Strategies.  ATS  investmrat  strategy 
options  were  evaluated  in  this  section.  Principal  results  included  an  economic  payoff  anal¬ 
ysis  that  was  based  on  four  weapon  system  case  studies.  Baseline  data  from  Section  4  were 
used  extensively  to  develop  the  individual  case  study  scenarios.  A  conunon  ATS  family 
investment  strategy  was  found  to  have  the  greatest  economic  benefits. 

Section  6,  DoD  Investment  Assessment  Analyas.  This  analysis  characterizes 
total  DoD  ATS  investments,  and  projects  the  savings  diat  will  result  from  a  new  DoD  com¬ 
mon  ATS  family  based  investment  strategy.  This  section  describes  in  detail  the  evaluated 
investment  strategy,  the  model  used  to  conduct  the  evaluation,  and  the  range  of  savings  that 
may  be  realized  if  the  new  ATS  investment  strategy  is  adopted.  TTiis  analysis  makes  exten¬ 
sive  irse  of  ATS  baseline  data  (Section  4)  and  ATS  investment  strategy  evaluation  results 
(Section  5). 
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Section  7,  Policy,  Process,  Funding  and  Organizations.  This  section  summarizes 
the  principal  ATS  policies,  and  describes  the  predominant  ATS  management  process  and 
funding  flow  for  DoD  organizations  found  to  be  involved  with  ATS  acquisitions.  Analysis 
addresses  dispersed  investment  decisions,  both  within  as  well  as  across  Service  and  weapon 
program  lines,  and  describes  how  distributed  ATS  management  has  been  a  major  contrib¬ 
uting  factor  to  DoD  ATS  proliferation. 

Section  8,  ATS  Capabilities  and  Applications.  This  section  describes  the  capabil¬ 
ities  of  DoD  designated  ATS  standard  families,  and  demonstrates  the  technical  feasibility 
of  following  the  investment  strategy  evaluated  in  Section  6.  ATS  test  capability  envelopes 
and  selected  weapon  system  testing  requirements  are  compared. 

Section  9,  ATS  Technology  Development  and  Evolution.  This  section  of  the 
report  provides  an  analysis  of  ATS  technology  development  and  evolution  influences  that 
have  a  direct  affect  on  ATS  used  by  the  Services.  Development  and  evolution  are  addressed 
from  six  different  perspectives:  ATS  industry,  standardization,  TPS  development  and  sup¬ 
port  environment,  vertical  testability,  Built-In-Testing  (BIT)  applications,  and  DoD  unique 
ATS  needs. 

To  help  manage  the  study  work  load  and  scope,  the  ESC  selected  15  weapon  sys¬ 
tems  as  representative  of  DoD-wide  programs  for  ATS  data  collection  and  in-depth  analy¬ 
sis.  Data  collected  from  these  IS  systems  were  used  to  establish  a  baseline  that 
characterizes  ATS  acquisition  costs,  scale,  and  investment  focus.  The  team  also  complied 
a  comprehensive  library  of  ATS  related  information  from  previous  studies,  DoD  and  indus¬ 
try  technical  perspectives,  and  Service  policies  and  standards. 

This  information  was  used  to  characterize  and  assess  the  magnitude  of  DoD’s  ATS 
investments,  to  evaluate  selected  investment  options,  and  to  characterize  economic,  tech¬ 
nical,  and  mission  benefits  of  an  improved  DoD  ATS  investment  strategy.  The  following 
sections  (Sections  4-9)  describe  the  major  findings  and  conclusions  in  the  context  of  the 
specific  analysis  where  they  were  observed.  Unique  analysis  details,  background  discus¬ 
sions,  and  study  constraints  are  described  where  they  are  pertinent  to  the  respective  analysis 
section. 
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4.  BASELINE  DATA 

4.1  BACKGROUND  DISCUSSION 

The  initial  phase  of  tiiis  study  effort  was  dedicated  to  the  collection  of  data  neces¬ 
sary  to  characterize  die  scale  and  scope  of  currmt  ATS  investments.  The  intent  of  this  initial 
phase  was  to  characterize  relationships  of  various  cost  elonents  in  the  acquisition  of  ATS. 
This  data  was  considered  fiindamental  to  the  study  and  ftnmed  the  baseline  for  all  invest¬ 
ment  related  analyses.  Because  of  the  broad  expanse  of  these  topic  areas,  the  ESG  directed 
die  study  team  to  use  existing  data  and  previous  studies  to  the  maximum  extent  practical, 
and  to  focus  new  data  collection  on  the  IS  selected  weapon  systems  shown  in  Figure  5. 


•  Abrams  (M-1)  * 

# 

Avenger  * 

•  F-16 

•  ACM 

AX  (A-12)* 

•  F-18  * 

•  AMRAAM* 

• 

Bradley* 

•  F-22  * 

•  AN/S(JQ-89 

• 

C-17* 

•  MK-50 

•  Apache/Longbow 

*  • 

F-15* 

•  MLRS 

*  iB  Depth  HiBt<ry^*roJccticiii 

] 

Figure  5.  Weapon  Sy^m  List  for  ATS  Study 

The  Executive  Steering  Group  included  die  new  Navy  AX  aircraft  on  the  weapon 
system  list,  and  suggested  that  data  from  the  cancelled  A-12  program  might  be  used  to  sup¬ 
plement  incoiiqilete  information.  To  simplify  references  to  the  Apache/Longbow  and  the 
AN/SQQ-89,  they  are  generally  referred  to  as  the  Apache  and  SQQ-89  respectively 
throughout  this  paper.  References  to  the  M-1  and  Abrams  tank  are  used  interchangeably 
diroughout  the  papa. 

Five  technical  interchange  meetings  (TIM)  were  conducted  during  1992.  The 
TIMs  were  used  to  present  the  data  requirements  and  to  support  the  data  collection.  Service 
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rqmsentatives  at  these  meetings  were  asked  to  coordinate  the  collection  and  delivery  of 
detailed  information  on  specific  ATS  types  used  or  planned  for  each  of  the  designated 
weapon  systems.  Copies  of  the  charts  used  to  identify  the  baseline  data  requirements  at  the 
TIMs  are  included  in  Appendix  A,  Data  Request  Formats  and  Details.  To  narrow  the  scope 
of  the  data  collection  work  load,  the  ESG  identified  systems  within  diis  group  as  the  most 
representative  of  current  DoD  ATS  needs.  These  systems  are  marked  with  an  asterisk  in 
Figure  5,  and  the  study  team  attempted  to  obtain  in-depth  historical  data  as  well  as  detailed 
projected  ATS  requirements  for  each  system. 

The  TIM  agenda  was  structured  around  the  overall  data  requirements,  and  each 
TIM  covered  a  different  set  of  ATS  topic  areas: 

•  TIM#1;  March  13  -  ATS  OiganizationaVManagerial  Baseline  &  Common  ATS 
Baseline 

•  TIM#2:  March  25  &  26  -  Technology  &  Related  Commonality  Issues/Trends 

•  TIM#3:  April  9  &  10  -  ATS  Acquisition  Process  &  Factory  Testing  Issues 

•  TIM#4:  April  21  &  22  -  Field  &  Depot  ATS  Commonality  Issues 

•  TIM#5:  May  4  &  5  -  Test  Requirements  &  Test  Development  Environments 

4.2  ATS  BASELINE  COST  SUMMARIES 

Tables  16,  17,  and  18  summarize  the  baseline  cost  data  collected.  More  detailed 
charts  are  presented  in  Appendix  E,  ATS  Data  Summaries.  In  general,  the  study  team 
found  ATS  acquisition  and  support  data  wav  not  easily  collected.  This  was  principally 
because  complete  libraries  of  ATS  acquisition  and  support  data  were  not  found  in  the  Ser¬ 
vices.  Since  this  may  represent  the  best  source  of  DoD  ATS  related  acquisition  and  support 
data,  the  following  paragraphs  summarize  the  study  teams  perception  of  its  quality  and 
completeness. 

Once  collected,  most  of  the  ATS  related  acquisition  (R&D  and  production)  data  was 
very  complete  and  was  very  consistent  in  form  and  substance  from  the  different  sources. 
The  three  exceptions  where  the  data  are  not  complete  are  marked  with  a  single  asterisk  in 
Tables  16  and  18.  The  study  team  was  unable  to  obtain  reliable  TPS  information  for  the 
MK-50.  All  of  the  data  collected  on  the  SQQ-89  was  considered  questionable  and  incom¬ 
plete  by  the  study  team.  The  C-17  weapon  system  acquisition  strategy  includes  the  devel¬ 
opment  of  ATS  as  part  of  a  support  capability  package,  and  as  such,  the  ATE  R&D  was  not 
separately  priced.  The  acquisition  data  reflected  what  the  DoD  paid  for  the  products  devel- 
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oped  and  delivovd.  The  populations  of  weapon  system  ATS  examples  for  which  data  were 
collected  must  inevitably  include  acquisition  instances  of  buy-ins,  ov»-runs,  sole-sources, 
etc.  No  attempt  was  made  to  differentiate  should-cost  estimates  from  actual  expenditures. 
The  authors  felt  this  was  a  reasonable  approach,  since  the  objective  was  to  evaluate  several 
ATS  investment  strategy  options  and  assess  relative  benehts  rather  than  determine  absolute 
costs. 

The  form  and  substance  of  the  support  data  varied  widely.  In  general,  the  ATS  relat¬ 
ed  support  costs  were  not  complete  and  could  not  be  used  except  to  indicate  trends.  The 
collected  depot  and  factory  ATS  information  focused  primarily  on  electronic  subsystems 
and  was  also  incomplete.  Characteristics  of  the  coUected  baseline  data  are  discussed  in 
more  detail  in  the  next  section. 


Table  16.  ATS  Summary  for  Selected  Navy  Weapon  Systems 


WEAPON 

SYSTEM 

ATS 

TYPES/ 

QTY 

TPS 

R&D 

($M) 

TPS 

PROD 

($M) 

ATE 

R&D 

($M) 

ATE 

PROD 

($M) 

A-12 

1/62 

183.6 

96.5 

24.8 

155.2 

AMRAAM 

2/27 

62.8 

18.1 

13.8 

58.4 

F-18 

9/453 

205.5 

203.4 

142.8 

759.8 

MK-50* 

1/10 

not  avail 

not  avail 

17.4 

19.7 

SQQ-89* 

5/14 

0.2 

not  avail 

not  avail 

1.7 

*  Indicates  data  not  comply 

Table  17.  ATS  Summary  for  Selected  Army  Weaptm  Systems 


WEAPON 

SYSTEM 

ATS 

TYPES/ 

QTY 

TPS  R&D 
($M) 

TPS 

PROD 

($M) 

ATE 

R&D  ($M) 

ATE 

PROD 

($M) 

ABRAMS 

4/202 

11.9 

31.9 

9.1 

182.7 

BRADLEY 

2/230 

13.6 

32.4 

10.6 

209.8 

APACHE 

6/47 

15.0 

33.5 

26.7 

239.9 

MLRS 

5/14.2 

19.0 

5.4 

1  4 

25.8 

AVENGER 

3/48.2 

9.3 

1.5 

17.2 

108.8 
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Table  18.  ATS  Summary  for  Selected  Air  Force  Weapon  Systems 


WEAPON 

SYSTEM 

ATS 

TYPES/ 

QTY 

TPS 

R&D 

($M) 

TPS 

PROD 

($M) 

ATE 

R&D 

($M) 

ATE 

PROD 

($M) 

ACM 

4/16 

27.4 

5.1 

16.3 

11.9 

C-17* 

2/12 

135.0 

32.5 

0.0 

50.1 

F-15 

24/402 

703.6 

366.7 

172.9 

1398.9 

F-16 

10/444 

251.9 

277.0 

431.9** 

1074.7 

F-22  (est) 

UNK/338 

150.4 

51.2 

39.3 

206.2 

*  Indicates  data  not  complete. 

**  Pius  additional  $338  million  product  improvement  deveb^ment  concurrent  with  iMxxtaic- 
tion  and  fielding. 

43  BASELINE  CHARACTERISTICS 

The  following  sections  discuss  in  detail  the  issues  and  findings  uncovered  during 
the  data  collection  and  baseline  data  analysis. 

43.1  DoD  ATS  Proliferation 

Analyses  revealed  proliferation  of  ATS  types  within  DoD,  and  subsequent  analyses 
in  this  paper  show  that  many  of  these  ATS  had  duplicative  capabilities.  The  effects  of  this 
proliferation  on  increasing  total  DoD  costs  have  been  recognized  by  multiple  sources.  A 
report,  requested  by  the  Assistant  Secretary  of  the  Navy  in  1976  [ATE  1976],  highlighted 
nuuiy  proliferation  concerns.  The  results  of  the  report  lead  to  a  cost  benefit  analysis  (CBA) 
and  competitive  development  of  CASS.  The  MATE  program  was  the  Air  Force’s  attempt 
to  reduce  proliferation  and  reduce  ATS  costs.  In  the  past  year,  four  DoD  IG  audits,  three 
GAO  audits,  and  a  HAC  special  investigation  all  criticized  ATS  proliferation  found  in  the 
Services. 

For  the  15  weapon  systems  reviewed,  over  100  unique  ATS  types  were  identified 
with  total  equipment  numbers  exceeding  2,300.  Tables  16,  17,  and  18  present  summaries 
of  the  types  and  quantities  observed.  Since  the  study  focused  on  direct  weapon  system  elec¬ 
tronics  test  equipment,  other  related  weapon  system  items  are  not  included  in  these  tables. 
For  example,  the  baseline  data  focused  primarily  on  onboard  system  electronics  and  did  not 
include  testing  requirements  for  other  equipnrent  such  as  F-16  pods  which  have  their  own 
unique  ATS  (i.e.,  ATS  for  the  LANTIRN  and  EW  -  AN/ALQ131).  In  another  example,  two 
Services  test  the  same  prime  equipment,  yet  use  different  automatic  test  equipment  (e.g., 
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the  Air  Force  and  Navy  use  diffoent  ATS  to  test  the  AIM-917M  Sidewinda*  Missile  Guid¬ 
ance  and  Control  Section). 


Recognizing  that  some  of  the  ATS  data  for  these  selected  systems  reflect  future  or 
projected  quantities,  four  of  the  Services’  current  front  line  weapon  systems  were  assessed 
separately.  Forty-seven  unique  ATS  were  identified  just  for  the  F-18,  M-1,  F-16,  and  F-15 
weapon  systems  alone.  The  47  ATS  types  were  found  to  be  used  in  field  (intermediate  or 
‘T”)  and  depot  level  tester  sq>plications,  and  in  some  cases  were  found  to  be  used  in  both 
applications. 

432  ATS  Acquisition  Cost  Factors 

ATS  costs  are  dominated  by  ATS  hardware  development  and  procurement  For 
example,  test  equipment  and  interface  device  hardware  R&D  and  production  costs  for  the 
F-18  are  over  80%  of  the  total  F-18  ATS  acquisition  cost  This  was  typical  of  the  weapons 
evaluated.  Emerging  programs  such  as  the  F-22  projected  higher  software  percentages 
(approximately  32  to  35%)  but  are  still  dominated  by  hardware  costs. 

The  magnitude  of  this  dominance  is  illustrated  in  Table  19.  Colunms  two  through 
five  are  percentages  of  principal  R&D  and  production  cost  categories  to  the  total  ATS 
acquisition  costs.  The  last  column  summarizes  the  approximate  percentage  of  hardware 
costs. 


Table  19.  ATS  Acquisition  Cost  Factors  for  Selected  Weapon  Systems 


WEAPON 

SYSTEM 

HW 

(%) 

ATSACQ 

(%) 

ATSACQ 

(%) 

ATS  ACQ 

(%>) 

ATSACQ 

(%) 

ABRAMS 

5 

14 

4 

77 

95 

BRADLEY 

5 

12 

4 

79 

95 

APACHE 

5 

11 

8 

76 

95 

F-18 

16 

16 

9 

59 

85 

F-15 

26 

14 

7 

53 

77 

F-16 

12 

14 

21 

53 

87 

Production  percentages  in  this  table  (both  TPS  production  and  ATE  production) 
account  for  recurring  engineering  tasks  that  are  almost  totally  hardware  related.  The  ATE 
R&D  percentages  include  a  mix  of  hardware  and  operating  system  software  costs.  Nomi- 


nally,  software  and  hardware  ATE  R&D  percentages  are  approximately  10%  and  90% 
respectively.  For  example,  the  F-16  AIS  development  costs  associated  with  the  control  and 
support  software  was  9%  or  $13.SM  of  $146.9  (FY93$)  [MATE  1990].  The  TPS  R&D  per¬ 
centages  also  include  a  mix  of  hardware  and  software  costs;  however,  the  relative  percent¬ 
ages  are  almost  reversed.  Data  from  both  the  Navy  [TPS  DEVEL  1992]  and  General 
Dynamics  [ATE  HISTORY]  imply  that  the  software  costs  are  approximately  87%  of  the 
total  TPS  R&D.  These  relative  percentages  were  used  to  estimate  the  hardware  (HW)  per¬ 
centage  column. 

Weapon  system  ATS  acquisition  cost  factors  were  also  evaluated  for  the  F-16  and 
F-IS  intermediate  level  and  depot  level  maintenance  equipment  The  hardware  costs  were 
still  dominant;  however,  the  software  cost  share  increased  significantly  for  the  depot  cases 
as  shown  in  Table  20.  This  condition  was  found  to  be  ATE  quantity  dependent  and  attrib¬ 
uted  to  die  fact  that  greater  numbers  of  each  ATE  type  are  found  at  the  intermediate  level 
(I-level)  maintenance  than  found  at  the  depot 


Table  20.  ATS  Acquisition  Cost  Factors  for  Selected  Maintenance  Locations 


WEAPON 

SYSTEM 

&MAINT. 

LEVEL 

HRRSH 

ATE  R&D 

ATE  PROD 

HW 

(%) 

ATS  ACQ 

(%) 

ATS  ACQ 
(%) 

ATS  ACQ 
(%) 

ATS  ACQ 

(%) 

F-15 1-Uvel 

21 

16 

5 

58 

81 

F-15  Depot 

37 

7 

13 

43 

67 

F-16I-Level 

8 

14 

22 

56 

91 

F-16  Depot 

50 

9 

19 

22 

55 

A  relatively  strong  grouping  was  observed  for  different  ATS  cost  component  ratios 
of  similar  weapon  system  applications.  Figures  6  and  7  illustrate  the  grouping  of  these 
ratios,  first  for  tactical  fighter  aircraft  avionic  systems  and  second  for  Army  systems.  The 
infCTcnce  from  these  tight  groupings  was  that  the  relative  distribution  of  principal  ATS 
costs  was  strongly  influenced  by  the  complexity  and  application  of  the  weapon  system 
being  tested. 

The  ATS  baseline  data  for  “real”  (i.e.,  not  projected)  systems  were  then  grouped  by 
six  weapon  system  category  types:  Weapons  Cairier/Tracked  Vehicle  (WCTV),  Missile- 
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Terrestrial,  Missile- Air  Launch,  Air- Army,  Air- Air  Force,  and  Air-Navy.  Table  21  summa¬ 
rizes  the  results  of  this  analysis. 


Table  21.  ATS  Cost  Apportionment  Ratios 


WEAPON 

SYSTEM 

TYPE 

TPS  DEVEL 

ATE  DEVEL 

1 

TPS  PROD 

ATE  PROD 

ATS  ACO 

ATS  ACQ 

ATS  ACQ 

ATS ACQ 

ATS  ACQ 

ATS  PROD 

WCTV* 

0.068 

0.057 

0.119 

0.756 

1.154 

Missile 

Terrestrial 

0.148 

0.108 

0.0367 

0.707 

1.375 

Missile 

Air  Launch 

0.375 

0.182 

0.108 

0.335 

2.257 

Air  -  Army 

0.048 

0.085 

0.106 

0.762 

1.153 

Air- AF 

0.204 

0.129 

0.139 

0.528 

1.500 

Air  -  Navy 

0.158 

0.096 

0.157 

0.589 

1.341 

*  Weapons  Carrier/Tracked  Vehicle 

Procurement  data  for  each  of  the  IS  selected  weapon  systems  also  was  collected  and 
analyzed  against  the  total  ATS  production  costs  for  each  of  the  weapon  system  category 
types.  Ratios  of  total  ATS  production  costs  against  total  weapon  system  procurement  costs 
were  developed  for  each  of  the  six  categories.  The  results  are  summarized  in  Table  22.  The 
ratios  in  Tables  21  and  22  were  used  to  estimate  DoD  ATS  investments  in  subsequent  anal¬ 
yses  which  are  discussed  in  more  detail  in  Section  6,  DoO  ATS  Investment  Analysis. 


Table  22.  Ratios  of  ATS  Production  Costs  To  Weapon  System  Production  Costs 


Missiles 

Aircraft 

WCTV* 

Terrestrial 

Air  Launch 

Army 

Navy 

AF 

0.031 

.006 

0.026 

0.027 

0.043 

0.017 

*  Weapons  Carrier/Tracked  Vehicle 


4.33  Annual  ATS  Support  Related  Costs 

Insufficient  data  was  available  to  rigorously  assess  ATS  operating  and  support 
costs.  However,  for  the  few  systems  where  data  was  accessible,  the  O&S  costs  were  sub¬ 
stantial.  The  best  data  acquired  implied  that  the  annual  intermediate  level  ATS  operations 
and  support  related  costs  exceed  $40M  and  $80M  for  the  F- 18  and  F- 16  weapons  systems 


req>ectively.  Data  for  the  other  systems  was  far  less  complete.  Table  23  illustrates  the  level 
of  accessible  data.  The  large  A’s  and  small  s’s  indicate  whether  the  information  collected  is 
believed  to  represent  all  or  some  of  each  expense  category. 

O&S  data  collection  was  han^red  by  multiple  factors,  the  most  significant  being 
that  this  type  of  data  was  not  purposefully  collected  by  any  of  die  Services.  Further,  the 
form  and  substance  of  the  data  across  support  related  expense  categories  varied  by  Service 
and  sometimes  by  application.  The  spares  costs  for  the  Navy  F- 1 8  were  considered  accurate 
and  complete  because  of  additional  accounting  and  resource  tracking  capabilities  imple¬ 
mented  to  handle  stock  funding  of  depot  level  repairables.  The  service  reports  and  sustain¬ 
ing  engineering  expense  categories  for  both  the  F- 16  and  the  F- 18  were  also  considered 
accurate  since  there  were  actual  expenditures  available  for  Service  review.  The  F- 16  and  F- 
18  data  reflects  the  average  funds  paid  to  General  Dynamics  and  Navy  Cognizant  Field 
Activities  respectively.  All  of  the  other  data  was  based  on  models  or  algorithms  developed 
by  the  Services. 

Although  not  well  represented  by  Table  23,  all  of  the  technical  experts  representing 
the  Services  at  the  HMs  indicated  that  costs  for  software  are  important  in  the  ATS  life  cycle 
beyond  acquisition.  As  an  ATS  “ages”  it  often  must  be  replaced  with  more  modem  ATS; 
the  cost  of  rehosting  test  software  often  dominates  this  decision.  The  Navy  is  investing  over 
$800  million  in  rehosting  software  from  six  aviation  testers  onto  the  CASS.  Although  this 
was  determined  to  be  cost  effective  because  of  the  cost  of  supporting  and  updating  six  dif¬ 
ferent  testers  over  time  instead  of  one  (CASS),  it  illustrates  the  need  for  reducing  the  cost 
of  rehosting  software  across  hardware  types.  The  Navy  has  developed  a  CASS  O&S  cost 
model  that  is  based  upon  earlier  analysis  work  and  historical  data.  It  is  being  used  to  project 
new  as  well  as  rehosting  TPS  costs,  including  projected  sustaining  requirements.  Although 
the  accuracy  of  the  model  was  not  assessed,  the  authors  observed  that  the  cost  categories 
were  very  comprehensive  and  included  all  of  the  categories  identified  in  this  study. 

4.3.4  Depot  ATS 

The  ATS  baseline  acquisition  data  for  depots  of  the  selected  systems  varied  in  terms 
of  accuracy  and  completeness.  Information  was  received  on  depot  tester  types,  their  capa¬ 
bilities,  and  the  commercial  versus  DoD-developed  ATS  demogrtqrhy.  In  general,  all  depots 
reported  some  degree  of  common  use  of  ATS  to  support  multiple  electronics  items.  The 
custom-built  ATS  were  found  to  be  dominant  in  the  depots  (75%  custom  ATS  for  most  elec¬ 
tronics  items),  many  of  which  were  targeted  to  one  weapon  or  system.  This  was  further  sup- 


Table  23.  Annual  ATS  Support  Related  Costs  in  Constant  FY93$(Millions) 
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ported  by  the  realization  that  much  of  the  depot  automatic  test  systems  is  refurbished 
automatic  test  equipment  that  has  made  a  transition  from  the  factory. 

The  relatively  small  depot  commercial  tester  percentage  (only  25%)  came  as  a  sur¬ 
prise  to  the  technical  experts  at  the  TIMs.  The  general  perception,  until  viewing  the  data, 
had  been  almost  the  reverse.  This  perception  difference  was  attributed  to  the  fact  that  the 
custom-built  automatic  test  equipment  were  generally  targeted  to  sets  of  unique  subsystem 
testing  needs  that  did  not  have  the  visibility  afforded  many  of  the  general  purpose  commer¬ 
cial  testers  that  were  in  frequent  demand.  However,  these  representatives  did  cite  numerous 
examples  of  conrunon  testing  requirements  that  were  best  met  with  COTS  automatic  test 
equipment,  some  of  which  arc  from  common  families  (or  extensions  of  a  family)  in  the 
commercial  sector.  The  Army  cited  a  recent  example  where  common  testing  requirements 
were  consolidated,  and  a  COTS  tester  was  competitively  acquired  as  a  common  tester  for 
general  digital  testing  needs  across  multiple  systems. 

The  Navy  and  Army  were  found  to  have  more  depot-common  automatic  test  sys¬ 
tems  because  of  the  wide  use  of  VAST  and  EQUATE,  their  two  previous  generations  of 
common  ATS  families.  For  the  data  reviewed,  the  greatest  reuse  of  intermediate  level  ATS 
in  the  depot  was  found  in  the  Navy.  This  high  level  of  reuse  was  attributed  to  the  fact  that 
the  Navy  generally  tends  to  move  their  repair  forward  (towards  the  operational  weapon  sys¬ 
tem)  as  much  as  possible.  Therefore,  the  ATS  testing  requirements  are  very  similar  and 
these  requirements  are  being  met  with  the  same  tester.  This  information  was  provided  at  the 
TIMs  or  as  a  result  of  TIM  action  items  [AF  DEPOT],  [ARMY  DEPOT  1992],  [NAVY 
DEPOT]. 

43,5  Factory  ATS 

The  study  team  was  unable  to  obtain  accurate  accounting  of  factory  ATS  invest¬ 
ments  for  the  selected  systems.  However,  there  is  evidence  that  the  DoD  investments  are 
sizeable.  Over  $10  billion  in  GFE  factory  test  equipment  are  accounted  for  in  DoD  records 
[GFE  ATE  1991].  Furthermore,  the  bulk  of  factory  ATS  is  contractor  developed  or  provid¬ 
ed,  paid  for  by  the  Government,  but  is  not  included  in  this  GFE  accounting. 

ATS  for  factory  manufacturing  in-process  and  acceptance  testing  are  typically 
developed  and  controlled  by  the  contractor.  Federal  Acquisition  Regulation  (FAR)  Part  45 
directs  that  equipment  to  perform  contracts  is  provided  by  the  contractor;  however  “special 
test  equipment”  (STE)  may  be  developed  under  a  contract  when  “advantages  to  the  Gov- 
enunent”  [FAR  1992].  In  application,  many  large  programs  pay  a  sizeable  STE  bill.  For 


5.  ALTERNATIVE  ATS  INVESTMENT  STRATEGIES 


The  authors  performed  in-depth  analyses  of  alternative  DoD  ATS  investment  strat- 
egy  options.  Five  investment  strategy  options  were  defined,  and  are  summarized  in  Table 
24.  Complete  option  definitions  are  presented  in  Appendix  C,  ATS  Investment  Strategy 
Options. 


Table  24.  Investment  Strategy  Option  Deflnitions 


OPTION 

DEFINITION 

I 

NO  CHANGE:  No  unified  coordinated  DOD  policy,  strategy,  ot  investments.  Some 
minimal  coordination  in  ATE  test  language  atea. 

n 

COMMON  ATS  FAMILIES:  A  DoD-wide  policy,  strategy,  and  related  investments  to 
^iply  common  ATS  families. 

m 

COMMON  ATS  SPECS:  A  DOD-wide  policy,  strategy,  and  related  investments  to  ap¬ 
ply  a  common  ATS  specification  set 

IV 

COMMON  SW  ENVIRONMENT:  A  DOD-wide  policy,  strategy,  and  related  invest¬ 
ments  to  apply  a  common  software  environment 

V 

WEAPON  SYSTEM  ENGINEERING  APmOACH:  A  DoD-wide  policy,  strategy,  and 
relied  investments  for  testability  based  on  a  wespon  system  engineering  approach 

Five  evaluation  categories  for  each  option  were  selected  by  the  study  team  and 
approved  by  the  Executive  Steering  Group  at  the  onset  of  the  analyses: 

•  Economic  Payoff, 

•  Operational  Commonality, 

•  Technological  Feasibility, 

•  Programmatic  Risk,  and 

•  Depot  Efficiency. 

Information  available  for  evaluating  the  ATS  investment  strategy  options  under  the 
Economic  Payoff  category  was  quantitative  in  nature  and  included  ATS  development,  pro¬ 
duction,  and  support  cost  data.  In  contrast,  information  available  for  evaluating  each  of 
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these  strategy  options  under  the  remaining  categories  was  more  qualitative  or  subjective  in 
nature. 

Therefore,  the  analyses  presented  in  this  section  of  the  paper  are  divided  into  two 
independent  analyses:  Section  S.l,  Economic  Analysis,  and  Section  S.2,  Qualitative  Anal¬ 
ysis. 

5.1  ECONOMIC  ANALYSIS 

The  economic  analysis  was  based  on  ATS  acquisition  and  O&S  cost  data.  The  ATS 
acquisition  costs  include  all  costs  associated  with  development,  production,  and  major 
modifications.  The  ATS  O&S  cost  data  collected  and  presented  here  are  less  complete  and 
include  ATS  operation,  maintenance,  training,  and  spares  cost  information.  Due  to  these 
data  completeness  differences,  the  economics  analysis  is  divided  into  two  sections:  ATS 
Acquisition  Costs,  and  Operations  and  Support  Costs. 

5.1.1  ATS  Acquisititm  Costs 

This  section  summarizes  a  major  analysis  effort  conducted  in  conjunction  with  the 
study.  Due  to  its  complexity,  it  is  divided  in  three  major  sections: 

•  Case-Based  Analysis  Approach, 

•  Cost  Factors  for  ATS  Investment  Strategy  Options,  and 

•  ATS  Acquisition  Case  Studies. 

The  first  section  provides  an  overview  of  the  approach  used  to  assess  ATS  acquisi¬ 
tion  cost  differences  for  the  selected  investment  strategy  options.  The  second  section  sum¬ 
marizes  the  onale  that  went  into  the  selection  of  option  cost  factors  used  in  the  case 
studies.  Tr  section  describes  four  weapon  system  ATS  case  studies  used  to  compare 
the  ATS  investment  strategy  options. 

The  first  four  investment  strategy  options  were  compared:  Option  I  (Baseline-No 
Change),  Option  n  (Common  ATS  Families),  Option  in  (Common  ATS  Specifications), 
and  Option  FV  (Common  Software  Environment).  Since  technology  advances  are  needed 
before  the  full  advantages  of  Option  IV  may  be  realized,  this  option  was  evaluated  from 
both  near-  and  long-teim  perspectives. 

The  investment  strategy  described  as  Option  V  (Weapon  System  Engineering 
Approach)  proposes  to  selectively  apply  aspects  of  the  other  four  options.  No  consistent 
approach  for  applying  Option  V  cost  factors  could  be  defined.  Without  a  consistent 


aiqvoach  across  each  scenario,  there  was  no  basis  for  when  to  apply  and  when  not  to  ^ply 
a  cost  factor.  Since  results  would  be  suspect  and  open  to  Ivoad  and  inconsistent  interpreta¬ 
tion  of  each  application  assumption.  Option  V  was  not  separately  evaluated.  However, 
details  presented  in  this  economic  analysis  section  show  syntergistic  benefits  of  combining 
acquisition  strategy  elements  of  Options  II  and  IV.  Therefore,  the  strategy  evaluated  in 
Section  6.0  of  this  study  selectively  applies  aspects  of  other  options,  and  is  similar  to  some 
aspects  of  Option  V. 

The  case  study  analyses  showed,  in  every  weapon  system  scenario  examined  (F- 1 8, 
F-16,  F-IS,  and  M-1),  that  a  sizable  acquisition  cost  avoidance  is  possible  by  eliminating 
multiple  unique  ATS  developments  and  procurements  and  instead  using  or  modifying  a 
selected  set  of  standard  tester  families.  As  reflected  in  Table  25,  projected  acquisition  sav¬ 
ings  averaged  25  to  35%.  The  savings  were  attributable  to  significant  reductions  due  to 
shared  development  costs,  production  economies  of  scale,  and  increased  tester  work  load 
efficiencies.  The  details  of  this  analysis  are  provided  in  the  following  three  sections. 


Table  25.  Case  Studies  ATS  Devdopment  and  Production  Costs 
(Constant  FY9^  in  Millions) 


SYSTEM 

CASE 

STUDY 

OPTION  I 
(Baseline - 
No  Change) 

OPTION  n 
(Common 
ATS  FamUies) 

OPTION  in 

(Common 
ATS  Specs) 

OPTION  IV 

(Common  SW 
Environment) 

Near  Term  [Long  Term] 

F/A-18 

$1,321 

$577 

$1,184 

$1209 

[$1129] 

M-1 

$394 

$300 

$364 

$390 

[$383] 

F-16 

$2,374 

$1,716 

$2,161 

$2,316 

[$2,140] 

F-15 

$2,501 

$1,602 

$2289 

$2,381 

[$2,012] 

S.1.1.1  Case-Based  Analysis  Approach 

A  two-step  approach  was  used  to  assess  the  four  selected  investment  strategy 
options. 

•  Step  1 :  ATS  investment  strategy  options  were  analyzed  and  assigned  ATS  cost 
factors. 


Step  2:  These  ATS  option  cost  factors  were  then  applied  to  specific  weapon  sys¬ 
tem  life  cycle  case  study  scenarios. 


Step  1:  Generic  ATS  expense  categories  were  selected  for  ATS  development  and 
production.  The  effects  on  each  ATS  expense  category  by  investment  strategy  Options  D, 
m,  and  rv  were  assessed  and  compared  against  the  Option  I  (Baseline).  The  result  was  a 
matrix  of  option  cost  factors  for  each  ATS  expense  categories.  These  cost  factors  were  each 
computed  as  a  ratio  against  its  respective  baseline  expense  category  that  had  been  assigned 
the  value  of  1.0.  This  produced  cost  factors  all  normalized  relative  to  the  baseline  option. 
Step  1  is  described  in  the  next  section.  Cost  Factors  for  ATS  Investment  Strategy  Option. 

Step  2:  Four  weapon  system  ATS  acquisition  life  cycle  scenarios  were  developed 
for  use  in  case-based  analysis.  The  four  weapon  systems  used  for  these  case  studies  were 
the  F-15,  F-16,  F-18,  and  M-1.  The  scenarios  for  the  weapon  systems  ATS  were  derived 
from  the  collected  baseline  data.  Actual  expenditure  data  was  used  where  possible  through¬ 
out  the  analysis.  The  option  cost  factors  developed  in  step  1  were  applied  to  the  ATS  acqui¬ 
sition  life  cycle  scenarios  for  each  of  the  weapon  systems.  Four  basic  assumptions  were 
applied  consistently  across  the  case  studies. 

•  The  baseline  data  accurately  summarizes  a  nominal  20-year  life  cycle  for  weap¬ 
on  system  ATS. 

•  Technology  risk  factors  were  applied  to  case  studies  in  order  to  normalize  sce¬ 
narios. 

•  Applying  sets  of  cost  factors  for  each  option  across  a  system  specific  scenario 
compensates  for  common  costs  that  were  not  determined,  and  neutralizes  the 
effects  of  different  maintenance  concepts,  missions,  reliability,  and  O&S  needs. 

•  The  capabilities  or  benefits  of  each  option  were  assumed  to  be  available  at  the 
onset  of  the  case  study  and  to  have  existed  throughout  a  system-specific  scenar- 


These  assumptions  are  discussed  in  more  detail  in  the  next  section.  Cost  Factors  for 
ATS  Investment  Strategy  Options. 

By  restricting  analysis  to  a  single  weapon  system  in  each  scenario,  the  effects  of 
component  reliability  and  applied  maintenance  concepts  were  minimized  across  each  ATS 
acquisition  strategy  case-based  analysis.  The  approach  permitted  the  analysis  to  focus  on 
the  effects  of  the  respective  options  for  each  expense  category  independently,  while  holding 
the  weapon  system  design  and  its  maintenance  and  support  concepts  fixed.  This  approach 
provided  a  process  for  evaluating  ATS  development,  acquisition,  and  support  strategies  for 
different  options  fairly  and  objectively  for  each  individual  weapon  system  scenario. 
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Because  of  die  number  of  assumptions  and  because  there  are  otha  details  ov«  an 
ATS  life  cycte  that  might  be  overlooked,  the  resulting  costs  should  not  be  considered  rep¬ 
resentative  of  actual  total  costs.  What  is  important  is  this  iqpproach  provides  a  very  good 
con^parison  of  the  relative  cost  benefits  across  ATS  investment  strategies  relative  to  Option 
I  (Baseline-No  Change). 

5.1.U  Cost  Factors  for  ATS  Investment  Strategy  Options 

Eight  ATS  expense  categories  were  selected  for  this  analysis:  ATE  Development, 
ATE  Acquisition,  TPS  Development,  TPS  Acquisition,  ATE  Support,  TPS  Support,  TPS 
New  Equipment  Re-Host,  and  TPS  Updated  Equipment  Re-Host  Each  of  these  categories 
are  explained  in  more  detail  below.  Individual  cost  factors  were  developed  and  applied 
against  each  of  these  ATS  expense  categories. 

The  current  ATS  acquisition  and  support  approach  was  defined  as  die  case  study 
baseline  (Option  I-No  Change).  All  baseline  ATS  factors  were  assigned  the  value  of  1.0, 
with  the  exception  of  TPS  Updated  Equipment  Re-host  The  rationale  for  this  difference  is 
presented  in  paragraph  a.  below. 

The  individual  cost  factors  in  Table  26  reflect  identified  expense  category  cost  del¬ 
tas  (or  fractional  portion)  attributed  to  the  respective  options  relative  to  Option  1.  The 
assumptions,  rationale,  and  source  data  behind  the  derived  cost  factors  are  described  in  the 
following  paragraphs  (a.  through  j.). 

a.  TPS  Updated  Equipment  Re-host.  The  ATS  expense  category  of  TPS 
Updated  Equipment  Re-host  was  defined  as  a  special  category,  and  was  to  be 
assigned  a  value  relative  to  the  TPS  Development  factor  within  the  respective 
option  category.  Since  all  of  the  TPS  documentation  and  the  coding  for  the  same 
basic  design  would  be  available,  the  non-recurring  engineering  work  load 
reduction  relative  to  re-hosting  on  totally  new  equipment  was  assumed  to  be 
66.6%.  This  value  was  assigned  based  on  a  review  of  the  percent  of  time  clas¬ 
sically  found  allotted  to  the  various  TPS  development  cycle  elements  (discussed 
later  in  paragrs^h  e.(3)  of  this  section).  Therefore,  the  ATS  Updated  Equipment 
Re-host  cost  factor  for  Option-I  was  assigned  a  value  of 0.33  (1.0-  0.67)  relative 
to  TPS  Development  If  actual  expense  information  was  available  under  Option 
I,  the  factor  was  assigned  the  value  of  1.0  and  actual  data  was  used. 


Table  26.  ATS  Option  Cost  Factors 


— 

EXPENSE 

CATEGORY 

opnoN-i 

(Baseline  - 
No  Change) 

OPTION-II 

(Common 

ATS 

Families) 

OPTlON-ni 

(Common 
ATS  Specs) 

OPTION-IV 

(Commcm  SW 
Environment) 

Near  Teim  [Long  Term] 

ATE  Devel 

1.00 

0.05 

0.87 

1.00 

[1.00] 

ATE  Prod 

1.00 

0.85 

0.93 

1.00 

[1.00] 

TPS  Devel 

1.00 

0.90 

0.90 

0.83 

[0.31] 

TPS  Prod 

1.00 

0.85 

0.93 

1.00 

[1.00] 

ATE  Sup 

1.00 

1.00 

1.00 

1.00 

[1.00] 

TPS  Sup 

1.00 

1.00 

1.00 

0.66 

[0.40] 

TPS  Re-Host 
(NewEq) 

l.OO 

1.00 

1.00 

0.25 

[0.25] 

TPS  Re-Host 
(Updated  Eq) 

0.33 

0.33 

0.33 

0.20 

[0.20] 

b.  Option  n  Cost  Factors.  All  of  the  Option  n  (ATS  standard  fanulies)  factors 
were  applied  against  typical  ATE  family  costs.  The  nominal  costs  of  CASS  or 
IFTE  were  used. 


c.  ATE  Development  Costs  (Options  n*IV).  Expenses  in  this  category  include 
the  development  costs  of  both  test  equipment  and  the  operating  system 
software. 

(1)  Under  Option  II,  almost  all  of  the  ATE  Development  costs  were  assumed  to 
have  been  previously  allocated  to  other  system(s).  In  other  words,  there  is  a 
large  percentage  of  these  development  costs  that  are  shared  across  multiple 
weapon  system  applications,  nie  prorated  share  of  these  development  cost 
could  have  been  included  under  this  category.  However,  for  this  analysis 
approach  they  were  included  in  the  case  studies  by  applying  a  common 
charge  to  each  of  the  Service-specific  case  studies  that  used  family  ATS  in 
a  scenario.  This  will  be  discussed  in  more  detail  in  Section  5. 1.1.3,  ATS 
Acquisition  Case  Studies.  The  basis  for  this  assumption  is  inherent  in  the 
common  family  investment  strategy,  and  is  also  representative  of  the  tech¬ 
nical  data  collected  on  the  CASS  and  IFTE  ATS  family  applications  in  the 
respective  Services.  To  account  for  any  unique  or  ^social  development 
requirements  needed  in  this  (or  other  future  system  testing  requirements),  an 
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additional  5%  of  the  original  ATE  family  development  cost  was  assigiwd  to 
this  expense  category.  The  5%  was  based  on  an  estimate  of  tl^  necessary 
costs  associated  with  an  assessment  of  testing  requirements  coverage  and 
capabilities,  the  identification  of  testing  enhancement  needs,  and  the  equiv¬ 
alent  of  a  tax  against  future  test  equipment  users  for  Pre-Planned  Product 
Improvement  (P^I)  engineering  and  design.  Therefore,  the  ATE  develop¬ 
ment  option  cost  factor  was  assigned  the  value  of  O.OS  against  the  original 
ATE  development  costs  and  thereby  being  representative  of  the  actual  ATE 
design  complexity. 

(2)  Based  on  historical  data  in  die  MATE  Effectiveness  Evaluation  Report 
[MATE  1990],  the  common  ATS  specifications  associated  with  the  MATE 
concept  pmnitted  approximately  90%  operating  system  code  reuse  and 
tqiproximately  4.1%  hardware  design  reuse  for  new  ATE  developments. 

The  test  equipment  hardware  design  reusability  percentage  was  derived 
firom  an  assessment  in  the  report  that  described  test  module  reusability.  The 
report  documented  an  average  MATE  module  development  cost  of  $  1 90.2K 
(FY90$).  The  report  went  on  to  ^ow  an  average  of  19  modules  per  test  sta¬ 
tion,  for  which  approximately  23%  were  non-MATE  modules  requiring  new 
development  (versus  available  off-the-shelf  MATE  modules),  and  ^proxi- 
mately  18%  of  all  the  modules  were  reused.  Combining  these  observations 
revealed  that  approximately  79%  of  the  modules  are  not  redesigned  per 
ATS.  Based  on  the  average  module  development  cost,  the  report  estimated 
$150K  (FY90$)  cost  avoidance  per  ATE  development,  or  when  related  to 
the  total  development  costs,  the  4.1%  savings. 

This  information  was  then  combined  with  the  estimate  that  the  test  equip¬ 
ment  hardware  and  the  operating  system  software  respectively  represent 
approximately  90%  and  10%  of  the  total  ATE  development  costs.  The  cost 
share  distribution  was  based  on  Air  Force  data  for  the  F-16  AIS.  The  com¬ 
bination  of  this  information  produced  a  cost  avoidance  percentage  of  12.7 
(90%  X  4.1%  +  90%  X  10%)  which  translates  to  an  Option-Ill  ATE  develop¬ 
ment  option  cost  factor  of  0.873  (l.(X)  -  0.127). 

(3)  The  common  software  environment  approach  under  Option-IV  was 
assessed  to  have  minimal  effect  on  the  ATE  development  costs;  therefore, 
the  option  factor  was  assigned  a  value  of  1.0. 
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d.  ATE  PitMhiction  Costs  (Ofrtions  IMV).  Expenses  in  this  category  include  the 
recurring  engimering  and  production  costs  to  acquire  the  hardware  and  any 
embedded  system  software. 

(1)  Given  the  near  100%  ATE  design  reusability  across  test  systems  for  Option- 
n,  the  potential  for  applying  a  recurring  engineoing  learning  curve  was 
assess^  as  high.  Both  GE  and  Grumman  reported  that  by  around  the  SOOth 
unit,  DoD  could  anticipate  an  estimated  30%  cost  reduction  attributed  to  a 
production  learning  curve.  This  translates  to  a  96%  learning  curve: 

Rate  of  reduction  ^plied  at  2'^  or  double  the  quantity,  whoe 

y=a*r‘’ 

yscost  of  unit, 

a=cost  of  1®*  unit, 

i=number  of  units,  and 

b=rate  of  reduction 

After  the  first  year  of  CASS  and  IFTE,  the  current  planned  production  will 
have  moved  along  this  learning  curve  sufficiently  to  realize  1S%  or  greater 
savings.  This  was  assessed  as  a  low  estimate  of  potential  savings,  since 
learning  curves  of  around  90%  are  common  for  electrical  and  mechanical 
commodity  assembly.  Although  actual  learning  curves  for  the  CASS  and 
IFTE  have  yet  to  be  determined,  the  F-I6A/B  AIS  ATS  experienced  a  pro¬ 
duction  learning  curve  of  71%  [CAIG].  Combining  this  15%  with  the  near 
100%  reusability  across  applications  results  in  the  assignment  of  an  ATE 
acquisition  Option-II  cost  factor  value  of  0.85  (1.0  -  0. 15). 

(2)  The  MATE  Effectiveness  Evaluation  Report  [MATE  1990]  documented  that 
“manufacturers  are  building  MATE  testers  made  up  of  about  31%  of  the 
modules  being  used  on  odier  programs.”  The  authors  asses that  this  per¬ 
centage  could  have  climbed  to  at  least  50%  with  a  more  aggressive  MATE 
implementation  approach.  Applying  the  same  value  used  in  paragraph  d.(l ) 
above  that  produced  a  15%  savings  over  similar  production  quantities  to  this 
50%  value  results  in  a  projected  7.5%  unit  cost  savings  that  may  be  applied 
across  the  full  analysis.  Therefore,  the  ATE  acquisition  Option-111  cost  fac¬ 
tor  was  assigned  a  value  of  0.925. 

(3)  The  common  software  environment  approach  under  Opfion-P/  was 
assessed  to  have  minimal  effect  on  the  ATE  Acquisition  costs;  tho'efore,  the 
option  factor  was  assigned  a  value  of  1.0. 


e.  TPS  Devdopmoit  Costs  (Options  n>lV).  Expenses  in  this  category  include 
the  non-recurring  oigineering  and  analyses  co^  to  develop  TPS  software, 
interface  test  adapter  (TTA)  hardware,  integration  and  testing,  and  operating 
documentation  and  instructions. 

(1)  The  MATE  Effectiveness  Evaluation  Report  [MATE  1990]  identified  a  total 
savings  of  9.7%.  This  TPS  development  savings  was  attributed  to  three  pri¬ 
mary  areas:  MATE  station  interface  commonality,  MATE  Control  and  Sup¬ 
port  Software  Standard,  and  the  standard  ATLAS  language  commonality. 
Therefore,  the  TPS  Development  Option-II  cost  factor  was  assigned  a  value 
equal  to  that  of  Option-Ill  or  0.903. 

(2)  By  their  very  nature,  each  of  the  three  primary  areas  discussed  in  paragraph 
e.(l)  above  would  be  completely  standardized  under  an  ATS  family  con¬ 
cept  Therefore,  for  this  study,  the  TPS  Development  Option-Il  cost  factor 
was  assigned  the  same  value  (or  0.903). 

(3)  Since  a  common  ATS  software  development  and  support  environment  does 
not  exist  and  since  some  development  efforts  will  be  required  to  achieve 
benefits  or  savings,  both  near-  and  long-term  sets  of  option  cost  factors  were 
developed.  The  near  term  was  designated  within  S  to  10  years,  while  the 
long-term  was  10  to  20  years.  The  data  provided  by  the  Navy  [TPS  DEVEL 
1992]  describing  the  percent  of  development  time  allotted  to  individual 
steps  of  the  TPS  development  cycle  were  assumed  to  be  representative 
(Table  27).  This  assumption  was  in  agreement  with  information  provided  by 
General  Dynamics  [ATE  HISTORY]  describing  TPS  development  produc¬ 
tivity. 


Table  27.  Percent  TPS  Development  Time  Allcrtted  to  Tasks 


TPS  DEVELOPMENT  CYCLE 
ELEMENT 

DEVELOPMENT 

TIME 

UUT  Analysis 

2% 

ATE  Analysis 

2% 

Functional  Flow 

10% 

Detailed  Test  Design 

15% 

ID  Design 

13% 

i 


Table  27.  Percent  TPS  Developmoit  Time  Allotted  to  Tasks  (Continued) 
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i 


i 
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TPS  DEVELOPMENT  CYCLE 
ELEMENT 

DEVELOPMENT 

TIME 

Coding/Compiling 

10% 

Integration 

46% 

Acceptance 

2% 

The  following  paragraphs  provide  the  assumptions  and  rationale  for  the  potential 
near-  and  long-term  cost  avoidance  benehts  against  each  of  the  development  cycle  ele¬ 
ments. 

UUT  Analysis:  Improvements  in  product  data  capture  coupled  with  improved  anal¬ 
ysis  tools  are  estimated  to  reduce  the  UUT  analysis  work  load  by  25%  in  the  near  term  and 
by  over  50%  in  the  long  term. 

ATE  Analysis:  Same  as  UUT  Analysis  above. 

Functional  Flow:  A  standardized  software  environment  such  as  ABBET  will  pro¬ 
vide  improvements  in  documentation  standards  that  facilitate  the  transport  of  UUT  design 
functional  flow  characteristics.  The  development  of  improved  tools  for  partitioning  func¬ 
tional  flow  characteristics  in  useful  formats  will  also  support  fault  isolation  and  fault  detec¬ 
tion  requirements  analysis.  Near-term  in^rovements  in  the  range  of  50%  are  assumed 
practical  for  digital  testing,  with  overall  long  term  TPS  development  benefits  in  excess  of 
75%  for  all  testing  domains. 

Detailed  Test  Design:  Automated  test  design  capabilities  are  evolving;  however, 
major  benefits  will  be  dependent  upon  standard  environments  and  new  tools  that  are 
designed  to  benefit  firom  these  standards.  The  authors  assumed  that  the  detailed  test  design 
element  work  load  will  decrease  by  approximately  33%  in  the  near  term  and  by  as  much  as 
66%  in  the  long  term.  This  growing  trend  is  supported  by  the  idea  that  standard  Hilaries  of 
tests  will  be  developed  and  available  for  reuse. 

Interface  Device  (ID)  Design:_The  authors  assumed  that  a  standard  software  envi¬ 
ronment  will  have  minimal  influence  in  the  near  term;  however,  the  growth  of  standard  test 
libraries  were  expected  to  promote  greater  ID  commonality  and  design  reuse.  These  obser¬ 
vations  lead  to  the  assumption  that  the  long-tram  benefits  will  be  in  the  50%  range. 

Coding/Compiling:  For  the  near  term,  the  authors  assumed  that  the  needed  tools 
and  libraries  would  not  be  in  place.  Therefore,  the  near  term  coding  benefits  of  ^proxi- 
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mately  10%  would  result  from  the  improved  electronic  availability  of  source  data  and  the 
limited  automated  TPS  genoration  for  some  digital  tests.  However,  major  benefits  in  the 
range  of  75%  would  be  available  in  the  long  term.  These  benefits  are  expected  to  be  the 
result  of  future  automated  tools  capable  of  electronically  receiving  the  detailed  test  design 
and  producing  reliable  reusable  code. 

IntegratitHi:  For  the  near-term,  this  area  will  continue  to  constitute  a  major  TPS 
development  work  load  driver.  Since  the  benefits  in  this  area  will  be  the  direct  results  of  the 
progress  and  implementation  made  in  the  preceding  development  cycle  elements,  the  near- 
term  benefits  were  assumed  to  be  in  the  10%  range.  However,  over  the  long-term,  the 
authors  assumed  that  the  work  load  associated  with  this  area  will  decrease  substantially 
because  of  the  improvements  made  to  the  entire  TPS  development  process.  Long-term  ben¬ 
efits  are  projected  to  range  around  75%. 

Acceptance:  Same  as  Integration  above. 

Summary  of  Option*IV  TPS  Development  Benefits:  The  TPS  Development 
Option-IV  cost  factor  was  assigned  a  value  of 0.832  for  the  near  term  and  0.305  for  the  long 
term.  Table  28  summarizes  the  i^proach  used  to  produce  these  option  faaors. 

Table  28.  Near  and  Long  Term  Optimi-IV  TPS  Development  Benefits 


TPS  Development 
Cycle  Elemoit 

Development 

Time 

Near*Term 

Benefits 

Long'Term 

Benefits 

UUT  Analysis 

2% 

X  25%  =0.5% 

x50%=  1% 

ATE  Analysis 

2% 

X  25%  =  0.5% 

x50%  =  l% 

Functional  Flow 

10% 

X  50%  =  5% 

X  75%  =  7.5% 

Detailed  Test 

Design 

15% 

X  33%  =  5% 

X  66%  =  10% 

ID  Design 

13% 

x0%  =  0% 

X  50%  =  6.5% 

Coding/Compiling 

10% 

X  10%=  1% 

X  75%  =  7.5% 

Integration 

46% 

X  10%  =  4.6% 

X  75%  =  34.5% 

Acceptance 

2% 

X  10%  =  0.2% 

X  75%  =  1.5% 

TOTAL 

100% 

16.8% 

69.5% 

f.  TPS  Production  Costs  (Options  Expenses  in  this  category  include  the 
recurring  engineering  and  production  costs  to  acquire  the  interface  devices  (ID) 
or  ITA  hardware  and  TPS  software  and  instructions. 

(1)  The  primary  benefits  for  Option-II  under  this  expense  category  would  fall 
under  potential  recurring  engineering  production  learning  curves  similar  to 
the  those  identified  in  paragraph  d.(l)  which  addressed  ATC  Production 
Costs.  Therefore,  the  TPS  Production  Option-D  cost  factor  was  assigned 
the  same  value  or  0.85. 

(2)  Similarly,  the  primary  benefits  for  Option-in  under  this  expense  category 
are  related  to  production  learning  curves  as  identified  in  paragraph  d.(2). 
Therefore,  the  TPS  Production  Option-Ill  cost  factor  was  assigned  the  same 
value  or  0.925. 

(3)  The  common  software  environment  approach  under  Option-IV  was 
assessed  to  have  minimal  effect  on  the  TPS  Production  costs;  therefore,  the 
option  factor  was  assigned  the  value  of  1.0. 

g.  ATE  Support  (Options  II-IV).  Expenses  in  this  category  include  direct  and 
indirect  logistics  and  sustaining  engineering  support  of  the  ATE  hardware  and 
any  embedded  operating  software.  The  differences  between  the  ATE  support 
costs  for  Options-I,  -0,  -HI,  and  -IV  were  assessed  to  be  minimal.  Therefore,  all 
ATE  Support  cost  factors  were  assigned  the  same  value:  1.0. 

h.  TPS  Support  (Options  II-IV).  Expenses  in  this  category  include  direct  and 
indirect  software  support  of  the  of  the  TPSs  and  the  BD/ITA  hardware. 

(1)  The  differences  between  the  TPS  support  costs  for  Options-I,  -II,  and  -III 
were  assessed  to  be  minimal.  Therefore,  TPS  Support  cost  factors  were 
assigned  the  same  value:  1.0. 

(2)  The  common  software  environment  approach  under  Option-IV  was  pro¬ 
jected  to  reduce  the  sustaining  engineering  and  software  analyses  by  50% 
for  all  continuing  support  efforts.  New  testing  capabilities  on  existing  equip¬ 
ment  were  assessed  to  be  of  similar  work  load  complexity  as  original  TPS 
development  or  approximately  17%  for  the  near  term  and  70%  for  the  long 
term  as  presented  in  paragraph  e.(3).  During  one  of  the  Technical  Inter¬ 
change  Meetings  (TIM),  the  Air  Force  assessed  the  work  load  split  between 
these  two  tasks  to  be  approximately  equal.  Combining  this  information 
results  in  a  projected  work  load  reduction  of  34%  (50%  x  50%  +  17%  x 
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50%)  for  the  near-term  and  60%  (50%  x  50%  +  70%  x  50%)  for  the  long¬ 
term.  Therefore,  the  TPS  Support  Option-IV  cost  factor  was  assigned  a 
value  of  0.66  for  the  near-term  and  0.4  for  the  long-term. 

i.  TPS  New  Equipment  Re-Host  (Options  II-IV).  Expenses  in  this  category 
include  the  reverse  engineering  and  development  of  replacement  TPSs  that  are 
re-hosted  on  replacement  or  alternate  ATE  which  represents  a  substantial 
change  from  the  original  hardware  and/or  operating  system  configuration. 

(1)  The  differences  between  the  TPS  Re-Host  costs  for  transferring  to  totally 
new  ATE  hardware  under  Options-I,  -II,  and  -HI  were  assessed  to  be  mini¬ 
mal.  Therefore,  TPS  New  Equipment  Re-Host  cost  factors  were  assigned 
the  same  value:  1.0. 

(2)  The  common  software  environment  approach  under  Option-IV  was 
assessed  to  reduce  the  reverse  engineering  and  analyses  work  load  by  75% 
for  all  TPS  being  re-hosted  on  new  equipment.  This  percentage  was  based 
on  the  conclusion  that  all  of  the  critical  information  will  be  in  electronic 
form  and  that  the  standardized  software  environment  will  permit  tools  that 
will  highly  automate  the  re-hosting  effort  including  the  integration  and 
acceptance  tasks.  Therefore,  TPS  New  Equipment  Re-Host  cost  factor  was 
assigned  a  value  of  0.25. 

j.  TPS  Updated  Equipment  Re-Host  (Options  Il-IV).  Expenses  in  this  category 
include  the  reverse  engineering  and  development  of  replacement  TPSs  that  are 
re-hosted  on  updated  ATE  which  represent  relatively  minor  configuration 
changes  to  the  original  hardware  and/or  operating  system. 

(1)  The  differences  between  the  TPS  Re-Host  costs  for  transferring  to  updated 
ATE  hardware  under  Options-I,  -II,  and  -ID  were  assessed  to  be  minimal. 
Therefore,  TPS  New  Equipment  Re-Host  cost  factors  were  assigned  the 
same  value:  0.33 

(2)  The  common  software  environment  approach  under  Option-IV  was 
assessed  to  further  reduce  the  reverse  engineering  and  analyses  work  load 
beyond  the  baseline  factor  of  0.33.  Since  the  re-host  for  new  equipment  was 
assigned  a  factor  of  0.25,  the  TPS  New  Equipment  Re-Host  cost  factor  was 
chosen  to  be  slighdy  less,  or  0.20. 
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5.1.1.3  ATS  Acquisition  Case  Studies 

The  following  sections  describe  the  general  approach  used  to  model  and  evaluate 
each  of  the  weapon  system  case  study  scenarios,  provide  detailed  discussions  and  results 
of  four  weapon  system  case  study  scenarios  (F- 1 8,  F- 16,  F- 1 5,  and  M- 1 ),  and  provide  anal¬ 
ysis  of  the  case  study  sensitivity  to  several  selected  assumptions  used. 

Table  29  identifies  the  four  weapon  systems  and  provides  an  overall  comparison  of 
the  case  study  results.  This  table  was  presented  earlier  as  an  overview  to  this  analysis  area 
and  is  repeated  here  for  continuity  in  this  section  (previously  Table  25). 


Table  29.  Case  Studies  ATS  Development  and  Production  Costs 
(Constant  FY93$  in  Millions) 


SYSTEM 

CASE 

STUDY 

OPTION  I 

(Baseline  - 
No  Change) 

OPTION  n 

(Common 
ATS  Families) 

OPTION  in 

(Common 
ATS  Specs) 

OPTION  IV 

(Common  SW 
Environment) 

Near  Term  [Long  Term] 

F/A-18 

$1,321 

$577 

$1,184 

$1,209 

[$1,129] 

M-l 

$394 

$300 

$364 

$390 

[$383] 

F-16 

$2,374 

$1,716 

$2,161 

$2,316 

[$2,140] 

F-15 

$2,501 

$1,602 

$2289 

$2,381 

[$2,012] 

5.1.U.1  Case  Study  Approach 

Four  models  that  represented  each  weapon  system  case  study  scenario  were  devel¬ 
oped  using  data  collected  earlier.  The  models  wCTe  intended  to  reflect  the  actual  ATS 
requirements  for  a  weapon  system  over  a  20-year  period  (e.g.,  actual  expenditure  data  col¬ 
lected  and  budgeted/funded  cost  projections).  These  models  were  used  as  the  baseline  or 
Option-I,  No  Change  scenario.  The  20-year  period  was  selected  as  a  representative  life 
cycle  for  the  purposes  of  the  case  studies.  The  other  options  were  evaluated  relative  to  this 
baseline.  The  option  cost  factors,  developed  in  the  previous  section,  were  then  applied  to 
appropriate  expense  categories  to  determine  the  relative  costs  for  each  option. 

The  following  assumptions  were  used  to  construct  the  four  weapon  system  case 
study  scenarios.  Assumptions  used  to  build  the  scenarios  were  based  upon  factual  sources 
except  as  specifically  noted.  These  four  systems  were  selected  because  they  are  represen¬ 
tative  of  the  major  DoD  weapon  systems  and  because  sufficient  historical  ATS  acquisition 
data  were  available  to  construct  representative  pictures  of  their  ATS  life  cycles. 


a.  The  baseline  data  was  assumed  to  accurately  summarize  a  20-year  life  cycle. 
The  information  used  to  construct  the  life  cycle  scenario  was  from  the  respec¬ 
tive  system  ATS  data  provided  by  the  Services  and  included  actual  expenses, 
budgeted  requirements,  and  known  projected  costs  as  reflected  in  the  Five-Year 
Defense  Plan  (FYDP).  This  expenditure  data  was  considered  very  representa¬ 
tive  of  actual  costs.  The  same  data  was  used  across  each  option  evaluation  for  a 
specihc  case  study  scenario  to  ensure  that  potential  data  omissions  or  errors 
would  not  invalidate  the  relative  differences  ultimately  computed. 

b.  One  or  more  families  of  testers  were  assumed  to  be  available  and  suitable  to 
meet  testing  and  mission  requirements  at  the  onset  of  the  specific  weapon 
system  (used  in  Option-II,  Common  ATS  Families).  Additional  factors  to 
compensate  for  any  differences  in  capabilities  and  tester  quantities  are 
addressed  in  the  next  paragraph. 


c. 


The  quantity  of  family  test  stations,  required  to  meet  specific  testing 
requirements,  was  based  on  known  (or  projected)  test  station  requirements  for 
a  replacement  family  of  testers  today  (1992).  To  compensate  for  any  ATS 
technology  differences  today  relative  to  the  capabilities  that  might  have  been 
available  at  the  start  of  the  specific  weapon  system  program,  a  technology  risk 
factor  was  applied.  The  technology  risk  factor  compensated  for  technology 
differences  by  adjusting  the  earlier  test  station  quantities  upward  linearly  up  to 
25%  over  a  13-year  period.  This  additional  cost  risk  was  reflected  as  a  delta 
procurement  cost  for  an  additional  quantity  of  ATE  (used  in  Option-II,  Common 
ATS  Families).  The  authors  recognized  that  the  technology  adjustment  could  be 
accommodated  from  two  perspectives:  (1)  increase  the  unit  acquisition  cost  or 
(2)  increase  the  quantity  of  testers  needed  to  perform  the  same  testing 
throughput  The  second  approach  was  selected  because  ATS  technology 
improvements  have  permitted  greater  throughput  capabilities  and  because  this 
approach  permits  graphical  representation.  Therefore,  as  applied  in  this  study, 
the  25%  was  based  on  a  subjective  estimate  of  potential  testing  throughput 
differences.  Further  evaluation  of  this  estimate,  described  in  the  Section 
5.1.1 .3.6,  Sensitivity  Analysis,  revealed  acquisition  cost  differences  of  less  than 
5%  relative  to  Option  II  and  2%  relative  to  the  baseline  for  the  F- 18  case  study. 


d.  Sets  of  specification  standards  were  assumed  to  be  in  place  throughout  the  life 
cycle  of  the  ATS  systems  represented  by  the  baseline  analyses.  Option  cost 
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factors  for  Option-in,  Common  ATS  Specs.,  were  then  applied  directly  to  each 
expense  category  of  the  baseline  (Option-I).  Similar  to  the  situation  in  the 
Option  II  evaluations,  these  standards  are  not  in  place  nor  are  they  defined  to 
represent  the  entirety  of  standards  needed  for  DoD  ATS  acquisition. 

e.  A  standard  ATS  software  development  and  support  environment  was  assumed 
to  be  in  place  throughout  the  life  cycle  of  the  system  as  presented  in  the  baseline 
analyses.  The  option  cost  factors  for  Option-IV,  Common  SW  Environment, 
were  then  applied  directly  to  each  expense  category  of  the  baseline  (Option-I). 
The  near-term  results  are  based  on  an  assumption  that  the  near-term  benefits 
were  in  place  throughout  the  weapon  system  life  cycle,  the  long-term  results  are 
based  on  an  assumption  that  the  long-term  benefits  were  in  place  through  the 
weapon  system  life  cycle.  The  existence  of  these  assumptions  permit  the 
evaluation  of  this  option,  in  an  approach  similar  to  that  used  in  Options  II  and 

m. 

f.  A  prorated  share  of  a  standard  ATS  family  development  cost  ($60  million)  was 
assigned  to  scenarios  that  used  a  tester  from  a  family.  This  $60  million  value 
was  based  on  a  very  conservative  assumption  that  an  ATS  family  of  the 
complexity  of  CASS  would  be  used  to  meet  3  weapon  system  testing  needs. 
This  area  is  also  discussed  further  in  Section  5. 1.1. 3.6,  Sensitivity  Analysis. 

g.  In  several  cases,  the  actual  reported  TPS  development  and  acquisition  for  the 
baseline  did  not  distinguish  between  the  relative  costs  for  circuit  cards  (also 
referred  to  as  shop  replaceable  units  or  assemblies  (SRU  or  SRA))  and  modules 
(also  referred  to  as  line  replaceable  modules  or  units  (LRM  or  LRU),  or  as 
weapon  replaceable  assemblies  (WRA)).  Therefore,  module  TPSs  were 
assigned  a  relative  cost  of  approximately  10  times  that  of  cards.  This  factor  was 
used  in  the  models  to  compute  weighed  TPS  development  and  acquisition  costs. 
This  relative  difference  in  card  to  module  cost  was  well  supported  throughout 
theTIMs. 

h.  For  the  ATS  development  costs  under  Option  II  (Common  ATS  Families),  the 
option  cost  factors  were  applied  to  the  actual  development  cost  of  an  ATS 
family  candidate  that  met  the  basic  technical  testing  requirements.  This  was  an 
exception  to  the  earlier  rule  that  cost  factors  were  applied  to  baseline  expense 
categories.  This  approach  change  was  justified  for  two  reasons:  (1)  by  applying 
the  5%  factor  to  the  original  development,  the  cost  more  closely  reflected  the 
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design  conq)lexity  of  the  ATS,  and  (2)  the  total  develt^ment  cost  for  the  ATS 
family  candidate  was  higher  (yet  distributed  across  multiple  systems)  than  the 
unique  ATS  systems  in  each  scenario. 

5.1.UJ  F/A-18  Scenario 

The  F/A-18  A/B  and  later  the  C/D  aircraft  are  supported  by  over  350  ATS  made  up 
of  8  unique  types.  The  Navy  plans  to  support  the  new  F/A-18  E/F  unique  avionics  with  31 
CASS  ATS  stations.  Approximately  4  years  later  the  Navy  intends  to  off-load  work  of  the 
8  older  ATS  stations  and  rehost  TPSs  on  64  additional  CASS  stations.  The  ATS  information 
used  to  formulate  the  baseline  is  presented  in  Table  30.  This  information  was  collected  by 
Navy  personnel  and  provided  specifically  in  support  of  this  study.  A  representation  of  the 


Figure  8.  Option  I,  No  Change  Scenario  for  the  F/A>18  Weapon  System 


This  scenario  representation  of  Option  I  in  Figure  8  shows  the  current  and  planned 
implementation  of  ATS  to  support  the  F/A-18.  Although  this  baseline  scenario  includes  a 
Navy  ATS  family  (CASS),  it  is  not  representative  of  a  case  study  scenario  that  implemented 
an  Option  II  (Conunon  ATS  Family)  at  the  onset  of  the  2()-year  life  cycle. 
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Table  30.  F-18  Baseline  Scenario  Information 


ATS 

Name 

qty 

Cards 

Modules 

ATE 

Oevel 

(SM) 

ATE 

Acq 

(SM) 

TPS 

Devel 

(SM) 

TPS 

Acq 

(SM) 

No. 

ITAs/ 

AIS 

Off 

Load 

ITAs 

AN/USM-470 
ATS  (VI) 

58,0  1 

! 

0 

43 

39.3 

290.7 

25.8 

21.9 

27 

9 

i 

AN/USM.446 

RSTS 

55.0 

21 

6 

10.0 

130.8 

20.0 

35.3 

27 

8 

AN/ASM-686 

lATS 

31.0 

0 

27 

11.3 

30.4 

32.9 

16.0 

27 

0 

AN/USM-629 

EOTS 

34.0 

0 

11 

21.5 

50.0 

5.6 

12.4 

11 

0 

AN/USM-392B 

DTBS 

14.4 

18 

0 

10.7 

5.6 

4.1 

l.I 

18 

■ 

AN/ASM-608  V 
IMUTSn 

49.0 

0 

3 

4.0 

22.8 

5.0 

4.0 

3 

3 

AN/USM-458C 

NEWTS 

28.2 

0 

7 

14.3 

14.4 

6.9 

■ 

■ 

■ 

AN/USM-484 

HTS 

88.0 

44 

1 

33 

45.7 

8.1 

23.0 

39 

0 

AN/USM-636 

CASSRF 

31.0 

106 

1 

i 

16 

93 

59.8 

4L1 

1 

28.1 

1 

21 

N/A 

The  Option-n  scenario  is  based  on  the  assumption  that  an  ATS  family  existed  at  the 
onset  of  the  F/A- 1 8  program.  The  algorithm  for  determining  the  number  of  ATS  family  sta¬ 
tions  required  at  the  start  of  die  program  was  based  on  the  fact  that  64  CASS  stations  are 
programmed  by  the  Navy  to  handle  the  F/A-18  A/B  and  C/D  testing  work  loads  after  the 
ATS  off-load.  This  number  was  then  adjusted  to  80  by  adding  in  the  25%  technology  risk 
factor  discussed  in  the  model  assumptions  from  the  previous  section.  At  ntid  way  through 
the  20-year  scenario,  the  ATS  family  was  assumed  to  under  go  a  mid-life  update.  At  that 
time  the  number  of  CASS  stations  required  to  support  the  F/A-18  A/B  and  C/D  was  read¬ 
justed  downward  to  the  actual  number  of  CASS  stations  needed  (64).  The  Option-II  sce¬ 
nario  model  includes  the  cost  of  updating  the  old  CASS  (the  80  stations  assumed  to  meet 
testing  needs  at  the  start  of  the  program)  to  the  64  current  CASS  stations.  The  cost  to  recon¬ 
figure  and  update  to  the  current  CASS  configuration  was  assumed  to  be  25%  of  the  produc¬ 
tion  costs  for  the  64  stations.  Figure  9  illustrates  this  scenario. 
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Figure  9.  Option  11,  ATS  Family  Scoiario  for  the  F/A-18  Weapon  System 


Both  Options  ID,  Common  ATS  Specifications,  and  Option  IV,  Common  Software 
Environment,  were  based  on  the  assumption  that  all  benefits  would  be  derived  from  either 
the  common  ATS  specifications  or  the  common  software  environment  respectively.  There¬ 
fore,  both  of  these  scenarios  are  based  on  the  assumption  that  at  the  mid-life  update,  the 
ATS  would  be  totally  replaced  one-for-one.  Figure  10  illustrates  the  scenarios  for  both 
options. 


Figure  10.  Option  HI,  Common  ATS  Specs  and  Option  IV,  Common  SW  Environ¬ 
ment  Scenarios  for  the  F/A-18  Weapon  System 
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The  relative  costs  for  all  ATS  changes  (developments,  acquisitions,  additions, 
updates  or  change  outs)  are  computed  based  on  the  relative  quantities  of  testers  needed 
according  to  the  developed  scenario  and  the  costs  as  adjusted  by  the  option  factors.  Sum¬ 
mary  of  the  F/A-18  case  study  are  presented  in  Table  31.  The  several  of  the  expensed  cat¬ 
egories  listed  in  this  table  are  composite  sununaries  of  the  ATE  and  TPS  development  and 
production  costs  for  the  event  identified  under  the  heading:  ATE  Mid-Life  Addition,  TPS 
(New  Dev  &  Prod),  ATE  Mid-Life  Update,  and  ATE  Change-Out. 

To  preserve  the  integrity  of  the  case  studies  and  for  consistency  across  scenarios, 
the  authors  decided  to  err  on  the  conservative  side.  For  example,  one  could  argue  that  the 
computed  TPS  production  cost  under  Option-ll  of  $102.9  million  is  too  high  and  would  be 
more  representative  of  projected  costs  if  reduced  to  $25  million.  The  value  in  Table  3 1  was 
derived  by  multiplying  the  TPS  acquisition  factor  firom  Table  26  (0.85)  with  the  baseline 
value  ($121.1  million).  This  approach  neglects  the  fact  that  the  number  of  ITAs  dropped 
down  from  8,298  (the  actual  total  number  identified  for  the  baseline,  Option-I  used  on  8 
ATE  types  at  multiple  locations)  to  1680  for  the  Option  D.  The  $25  million  value  was  com¬ 
puted  based  on  the  reduced  number  of  ITAs. 

In  another  example,  the  assigned  prorated  share  of  ATS  family  development  costs 
was  $60  million.  This  would  imply  that  an  ATS  family  of  the  complexity  of  CASS  will  only 
be  used  on  approximately  three  weapon  systems.  Although  it  makes  little  difference  in  the 
F/A-18  set  of  scenarios,  this  assumption  represents  further  savings  in  the  other  weapon  sys¬ 
tem  case  studies  to  be  discussed  in  the  next  section. 

5.1.13J)  F-16  Scenario 

The  F-16  weapon  systems  are  supported  by  four  generations  of  ATS  at  the  interme¬ 
diate  maintenance  level.  The  four  ATS  generations  are  closely  tied  to  the  aircraft  series  as 
indicated  by  their  names  and  these  ATS  were  acquired  as  the  weapon  system  series 
evolved:  A/B  Avionics  Intermediate  Shop  (AIS),  C/D  AIS,  Advanced  Computer  AIS,  and 
Improved  AIS  (lAIS).  Several  different  specialized  depot  ATS  were  also  acquired  over  the 
life  of  the  weapon  system.  Figure  11  illustrates  the  quantities  and  time  frame  that  the  dif¬ 
ferent  F-16  ATS  were  introduced.  The  conditions  illustrated  in  this  figure  represent  Options 
1,  in,  and  IV  weapon  system  scenarios  (No  Change,  Common  ATS  Specs,  and  Common 
SW  Environment  respectively).  The  baseline  scenario  information  used  to  develop  this  fig¬ 
ure  is  presented  in  Table  32. 


Table  31.  F*18  Case  Study  Summary  (FY93$-MiUion) 


EXPENSE 

CATEGORY 

! 

OPTION.I 

(No 

Change) 

opnoN-n 

(Common 
ATS  j 

Family) 

OPTION-ni 
(Common 
ATS  Specs) 

OPTION-rV 
(Common  SW 
Environment) 

Near  Term  /  Long  To'm 

ATE  Development 

114.4 

8.9 

99.9 

114.4 

114.4 

ATE  Production 

590.3 

131.2 

546.0 

590.3 

590.3 

TPS  Development 

108.4 

97.9 

97.9 

90.0 

33.1 

TPS  Production 

121.1 

102.9 

112.0 

121.1 

121.1 

SUBTOTAL 

934.2 

340.9 

855JI 

915it 

858.9 

ATE  Mid-Life 
Addition 

69.1 

59.7 

59.8 

65.5 

65.5 

TPS  (New  Dev  & 
Prod) 

69.2 

61.0 

33.3 

32.0 

17.7 

SUBTOTAL 

138J 

120.7 

93.1 

97J 

83.2 

ATE  Mid-Life 

Update 

- 

22.6 

• 

- 

- 

TPS  Rehost 

- 

32.3 

- 

- 

- 

SUBTOTAL 

0.0 

54.9 

0.0 

0.0 

02) 

ATE  Change-Out 

132.1 

- 

110.2 

120.0 

120.0 

TPS  Rehost 

64.6 

- 

65.2 

15.2 

6.8 

SUBTOTAL 

188.0 

0.0 

175.4 

135J 

126A 

Prorated  Share  of 
ATS  Family 

60.0 

60.0 

60.0 

60.0 

60.0 

TOTAL 

1,320.5 

576.5 

1,184J 

138.5 

1,128.9 

In  contrast  with  the  F/A-18  case-based  analysis,  one  figure  is  required  to  represent 
these  three  option  scenarios  rather  than  two.  This  is  attributable  to  the  fact  that  Options-III 
and  rv  of  the  F/A-18  analysis  represented  a  special  situation  that  made  it  necessary  to  build 
a  separate  set  of  scenarios  where  a  common  ATS  family  would  not  be  replacing  the  eight 
existing  automatic  test  systems.  For  fiiis  F-16  case-based  analysis,  however,  the  represen¬ 
tative  cost  factors  for  Options-1,  m,  and  IV  are  all  applied  to  the  same  quantity  of  ATS. 

To  develop  an  Option-n  scenario,  an  ATS  family  candidate  that  was  consistent  with 
the  assumptions  used  for  the  F/A-18  case  study  and  met  the  basic  technical  testing  require- 
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Figure  11.  F*16  Weapon  System  ATS  Evaluation  Scenario  for 
Optiims  1,  m,  and  IV 

merits  of  die  F-16  systems  needed  to  be  selected.  CASS  RF  (Radio  Frequency)  station  type, 
augmented  with  an  electro*optical  testing  (EO)  capability,  was  chosen  as  the  family  candi¬ 
date.  The  F-16  AIS  consists  of  4  different  station  types:  Computer  Inertial,  RF,  Display 
Indicator,  and  Processor/Pneumatic.  Four  CASS  RF  stations  plus  1  EO  augmentation  were 
assumed  to  meet  testing  throughput  requirements  of  the  full  F-16  AIS.  Therefore,  for  the 
purposes  of  this  scenario  option,  the  ATS  family  candidate  was  priced  at  $9.25  million 
(FY93$)  or  4  times  the  cost  of  the  CASS  RF  station  ($2.06  million)  plus  $1  million  for  the 
EO  augmentation.  This  assumption  was  justified,  since  other  analysis  revealed  that  2  or  3 
CASS  RF  stations  would  more  than  adequately  handle  the  current  F-16  AIS  (4  stations) 
testing  work  load  (e.g.,  a  station  reduction  of  1  or  2  per  location). 


The  technology  risk  factor  was  used  to  adjust  the  quantity  of  candidate  family  ATS 
intended  to  replace  the  A/B  AIS,  C/D  AIS,  Advanced  Computer  AIS,  and  Depot  ATE  (lin¬ 
eally  prorated  by  the  number  of  years  proceeding  the  current  CASS  based  technology). 
Since  the  F-16  lAlS  was  a  significant  departure  from  the  other  AIS,  it  was  treated  in  the 
scenario  as  a  new  ATS  family  “downsized  tester”  candidate.  Figure  12  illustrates  the  F-16 
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ATS  family  (Option-Il,  Common  ATS  Families)  scenario  used  to  evaluate  the  acquisition 
costs. 

Table  32.  F<16  Baseline  Scenario  Information  (FY93$>Millions) 


ATSNaaw 


A/BAIS 


C/DAIS 


Factn 


Dust 


QTY 

Cards 

Modules 

ATE 

Devel 

ATE 

Acq 

TPS 

Devel 

TPS 

Acq 

No. 

ITAs/ 

Him 

iHim 

IHilulBi 

ATS 

26 

0 

52 

146.9 

267.5 

71.2 

57.0 

42 

18 

0 

69 

129.7 

235.9 

20.4 

61.8 

52 

40 

0 

76 

29.4 

425.1 

20.8 

112.9 

52 

16 

0 

22 

89.4 

102.4 

41.3 

27.5 

18 

224 

0 

1.1 

5.3 

25.5 

5.8 

132 

13 

218 

0 

17.1 

2.9 

33.2 

5.6 

125 

11 

42 

0 

6.7 

21.5 

12.1 

3.0 

27 

107 

0 

11.5 

11.8 

22.6 

1.9 

36 

1 

116 

0 

0.0 

02 

4.1 

03 

54 

8 

46 

0 

0.1 

2.1 

0.7 

12 

46 

CASS  Based  D^t  ATE 


Downsized  AIS  Family 


CASS  Based  AlS’s 
(Quantity  Adjusted) 


(Quantity  Adjusted) 


(Quantity  Adjusted) 


1 

0 

1 

4 

1  1  1 

8  12  16 

1 

20 

Years  After  Start  of  Program 

Figure  12.  Option  11,  ATS  Family  Scenario  for  the  F-16  Weapon  System 
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The  F*16  case  study  summary  is  presented  in  Table  33.  The  approach  used  through¬ 
out  the  scenario  was  identical  to  that  used  for  the  F/A-18;  however,  the  presentation  of  the 
information  differs  slightly.  The  F-16  depot  ATS  information  was  identified  separately  on 
this  table.  The  information  is  displayed  differently  because  of  the  maintenance  concept  dif¬ 
ferences  and  in  order  to  reflect  the  mid-life  additions.  For  example,  the  Air  Force  and  Navy 
concepts  for  levels  of  repair  and  maintenance  are  different.  At  the  I-level  maintenance 
shop,  the  Air  Force  generally  fault  isolates  to  a  failed  SRU  by  testing  LRUs.  The  suspect 
SRUs  are  returned  to  the  depot  for  further  fault  isolation  and  repair.  Whereas,  the  Navy 
generally  provides  testing  down  to  the  equivalent  of  the  SRU  (called  SRA  by  the  Navy). 
This  results  in  different  Air  Force  ATS  to  meet  depots  and  field  testing  needs. 

Due  to  the  time  phasing  of  the  ATS  to  meet  the  evolving  aircraft  series  testing 
requirements,  it  was  convenient  to  separate  out  die  various  ATS  mid  life  additions. 

Since  CASS  was  used  as  the  ATS  family  candidate,  the  F/A-18  average  computed 
TPS  development  and  production  costs  for  cards,  modules,  and  ITAs  were  used  for  consis¬ 
tency.  These  were  the  same  values  used  in  the  F/A-18  case  study  scenario.  This  approach 
was  conservative  because  the  Navy  includes  a  greater  share  of  the  O&S  related  fielding 
tasks  under  the  acquisition  cost  of  the  TPS  (i.e.,  training,  documentation,  manuals,  etc.). 

The  prorated  cost  share  for  ATS  families  (Option-II,  Common  ATS  Family)  under 
the  F-16  scenario  was  greater  than  that  for  the  F/A-18.  In  the  F-16  scenario,  the  authors 
assumed  there  were  two  sets  of  different  ATS  families  used  over  the  life  cycle:  (1)  the  AIS 
replacement  and  (2)  the  downsized  tester.  Therefore,  the  prorated  share  assigned  was  to  be 
2  times  $60  million  or  $120  million. 

The  data  in  Table  32  includes  the  results  of  extensive  discussions  regarding  how  to 
address  the  additional  F-16  weapon  system  production  funds  (Air  Force  budget  category 
3010  -  aircraft  production  funds)  paid  to  the  ATS  contractor  for  a  range  of  P^I  equivalent 
efforts  over  the  ATS  life  cycle.  An  agreement  was  reached  by  the  F-16  System  Program 
Office  (SPO)  and  the  authors  that  these  funds  fell  in  the  following  three  categories:  service 
reports,  sustaining  engineering,  and  maintenance. 

Service  Reports  (SRs).  Service  reporting  is  the  process  by  which  users  identify 
problems  or  deficiencies  with  fielded  systems  to  the  procuring  activity  prior  to  system  pro¬ 
gram  management  responsibility  transfer  (PMRT).  For  the  F-16  SPO,  service  reporting  is 
a  separate  Contract  Line  Item  Number  (CLIN)  under  the  AIS  multiyear  contracts  that 
requires  General  Dynamics  (GD)  Fort  Worth  to  maintain  a  system  for  receipt  of  SRs,  con- 
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Table  33.  F>16  Case  Study  Summary  (FY93$-Millioii) 


EXPENSE 

CATEGORY 

OPnON-I 

(No 

Change) 

opnoN-n 

(Common 

ATS 

Family) 

opnoN-m 

(OMnmon 
ATS  Specs) 

OPTION-IV 

(Common  SW 
Environment) 

Near  Term  /  Long  Term 

ATE  Deveiopment-A/B 

146.9 

8.9 

128.3 

146.9 

146.9 

ATE  Production  -A/B 

267J 

255.6 

247.4 

267.5 

267.5 

TPS  Development  -A/B 

71.2 

72.3 

64.3 

59.1 

21.7 

TPS  Production  -A/B 

57.0 

40.1 

52.7 

mEm 

57.0 

SUBTOTAL 

542.6 

3764 

492.7 

5304 

493.1 

ATE  Mid-Life  Addition 

C7D 

365.6 

174.0 

331.4 

365.6 

365.6 

Adv  Computer 

454.5 

364.1 

418.9 

454.5 

454.5 

lAIS 

191.7 

134.8 

172.7 

191.7 

191.7 

TPS  (New  Dev  &  Prod) 

C/D 

82.2 

58.0 

754 

78.8 

68.0 

Adv  Computer 

133.7 

89.9 

1234 

130.1 

1194 

lAIS 

68.8 

414 

62.8 

61.8 

40.1 

SUBTOTAL 

1 

1496.5 

8624) 

1,1844 

14824 

1439.1 

Depot  ATS 

ATE  Devel 

^65 

53.4 

31.9 

36.5 

36.5 

ATE  Prod 

43.8 

774 

40.5 

43.8 

43.8 

TPS  Devel 

98.1 

104.7 

88.6 

81.5 

29.9 

TPS  Prod 

Yl.l 

121.7 

16.4 

nn 

17.7 

SUBTOTAL 

196.1 

3574) 

1774 

1794 

127.9 

Prorated  Share  of  ATS 
Family 

0.0 

120 

0.0 

0.0 

0.0 

P^I  Sustaining  Eng. 

338.3 

0.0 

306.7 

323.9 

279.5 

TOTAL 

24734 

1,715.9 

2,1614 

24164 

2,1394 
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duct  analysis  to  identify  die  cause  of  the  SR  i»'oblem,  and  suggest  a  corrective  action.  SR 
costs  are  apportioned  equally  between  TPSs  and  the  ATE. 

Sustaining  Engineering.  Sustaining  engineering  is  the  vehicle  for  implementing 
corrective  actions  recommended  by  SR  investigations.  It  is  also  a  separate  CLIN,  and 
includes  activities  such  as  engineering  change  proposal  preparation,  software  coding,  inte¬ 
gration,  software  reproduction.  Technical  Change  to  Technical  Order  (TCTO)  preparation, 
and  semi-annual  updates  to  station  control  and  support  software.  These  costs  are  also  even¬ 
ly  split  between  TPSs  and  ATE. 

Maintenance.  Maintenance  covers  all  the  direct  and  indirect  costs  involved  with 
maintenance  and  repair  of  GFE  ATE  and  TPSs  in  the  care  of  the  contractor.  GD  Fort  Worth 
currently  maintains  one  of  each  configuration  of  the  F- 16  AIS,  three  pre-production  version 
A/B  AISs,  and  a  large  number  of  TPSs.  These  items  are  used  for  sustaining  engineering, 
modification  integration  and  testing,  and  aircraft  production  line  check-out.  Maintenance  is 
also  a  separate  CLIN. 

A  sununary  of  the  sustaining  engineering  and  SR  costs  used  to  estimate  the  Option- 
I  P^I  sustaining  engineering  line  of  Table  33  are  presented  in  Table  34.  For  Option-II  the 
authors  assumed  no  contractor  sustaining  engineering,  and  for  Options-ni  and  FV  the 
authors  applied  the  average  of  the  appropriate  cost  factors  to  the  baseline  (Option-I)  value. 
Data  was  provided  by  the  F-16  SPO  for  years  1980  to  1995:  years  1980  through  1985  rep¬ 
resent  estimated  totals,  years  1986  through  1989  are  actual  costs,  years  1990  through  1993 
are  prenegoatiated  costs,  and  years  1994  and  1995  are  estimated  costs.  To  address  a  full  20- 
year  ATS  life  cycle,  averages  were  computed.  The  P^I  development  and  production  sustain¬ 
ing  engineering  was  assumed  to  include  all  of  the  reported  sustaining  engineering  column 
and  half  of  the  maintenance. 

The  results  of  this  case  study  clearly  indicate  that  the  ATS  family  scenario  acquisi¬ 
tion  strategy  produces  the  greatest  savings  undo*  the  stated  assumptions.  Given  these 
results,  the  authors  decided  to  expand  this  case  study  evaluation  to  compare  the  baseline 
life-cycle  development  costs  of  the  F-16  ATS  with  similar  development  costs  for  multiple 
applications  of  a  CASS  and  IFTE  ATS  families.  This  expanded  evaluation  was  based  on 
the  assumption  that  two  additional  weapon  system  testing  needs  would  be  added  each  year 
to  the  ATS  family  candidates  (one  for  each  CASS  and  IFTE).  Only  ATE  development  costs 
were  included  in  this  expanded  evaluation. 


Table  34.  Summary  of  (Actual  and 
Used  to  Fund  Continuing 


Projected)  F>16  Aircraft  Production  Resources 
ATS  Development  (FY93$-Millions) 


Year 

Sustaining 

Engineering 

Service 

Reports 

Maintenance 

Equivalent 
Continuing  P^l 

1980 

16.7 

1.7 

4.2 

18.8 

1981 

11.6 

1.2 

4.6 

13.9 

1982 

7.3 

1.0 

5.1 

9.9 

1983 

7.0 

0.7 

5.6 

9.8 

1984 

10.2 

1.0 

6.1 

13.2 

198S 

13.1 

2.0 

7.2 

16.7 

1986 

28.9 

3.8 

8.8 

33.3 

1987 

14.7 

2.0 

5.2 

17.2 

1988 

19.3 

2.5 

8.3 

23.4 

1989 

10.7 

1.3 

5.1 

13.3 

1990 

14.9 

0.8 

4.5 

17.1 

1991 

13.9 

0.8 

4.4 

16.2 

1992 

14.7 

0.8 

5.1 

17.3 

1993 

17.5 

0.8 

6.0 

20.5 

1994 

14.5 

0.9 

6.1 

17.6 

1995 

9.4 

0.8 

6.1 

12.4 

Subtotal  (16yr) 

224.4 

21.9 

92.4 

270.7 

Avg  Per  Year 

14.0 

1.4 

5.8 

16.9 

4- Year  Delta 

56.1 

5.5 

23.1 

67.6 

Est  20  Year  Total 

280.5 

27.4 

USJ 

338J 

The  same  5%  of  the  original  development  cost  for  each  new  tester  application 
would  be  applied.  In  addition,  the  need  for  a  mid-life  update  at  the  10  year  point,  with  an 
adjustment  to  the  development  cost  by  an  additional  25%,  was  assumed.  Results  are  illus¬ 
trated  in  Figiue  13  and  show  that  the  combined  ATE  development  costs  of  over  30  ATS 
family  applications  may  cost  less  than  the  development  of  the  F- 16  automatic  test  equip¬ 
ment  that  supported  two  basic  system  types  (A/B,  and  C/D).  Bodi  the  CASS  and  IFTE  in 
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this  example  are  assomed  to  support  the  core  systems  for  which  they  were  originally  devel¬ 
oped  plus  16  additional  systems  at  the  conclusion  of  the  16  years. 


Camnlative  R&D  Cost  Profile 
Assumes  Mid-Life  Update  for  CASS  &  IFITE  at  10  Years 
Assumes  2  Addidonal  Weapon  System  Testing  Requirements  Added  Each  Year 

(1  CASS  &  1  DTE) 

700 

S  600 

I  500 
s 

m  400 

tL 

I 

^  200 
100 
0.00 

1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16 

Years  From  First  FieMing 

Figure  13.  Development  Cost  Profile  Comparisons  of  a  System  Unique  (F*16) 
and  Two  Multiple  System  Application  ATE  (CASS  &  IFTE) 

5.1.1.3.4  F<15  Scmario 

The  F-IS  weapon  systems,  at  the  intermediate  maintenance  level,  are  supported  by 
a  combination  of  ATS  that  spans  the  life  of  the  aircraft  and  several  generations  of  new  ATS. 
The  newest  ATS  generation  included  as  a  part  of  this  case  study  is  a  proposed  “Downsized 
Tester”  (DST)  that  is  in  source  selection  as  the  study  continues.  Like  the  F- 16  weapon  sys¬ 
tem,  several  different  specialized  depot  ATS  were  also  acquired  over  the  life  of  the  weapon 
system.  Figure  14  illustrates  the  quantities  and  time  frame  that  the  different  F-IS  ATS  were 
introduced.  The  conditions  illustrated  in  this  figure  represent  Options-1,  m,  and  IV  weapon 
system  scenarios  (No  Change,  Common  ATS  Specs,  and  Common  SW  Environment 
respectively).  The  baseline  scenario  information  used  to  develop  this  figure  is  presented  in 
Table  35  and  Table  36. 


AADT*!  mem  MOD  TEST  STA 

400-1  DADTS^v  CGDTS-TO^^ 
MADTSCS>A. _ _^F 


ALM-205,  ^  205A,  .2§I 


AN/ASM^7. 


MTS  (MSIP) 
ANTENNA  AB  (MSIP) - ^ 


METS. 


ARTS. 


AN/ALM.173 


"1  AN/GSM.228 

0-1 - , - ^ ^ - , - 1 

0  5  10  15  20 

Years  After  Start  of  Program 


Figure  14.  F-15  Weapon  System  ATS  Evaluation  Scenario  for  Options  I,  m,  and  IV 
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DSTS  (est.)  61  (all)  -  97  2  84.9  -  82.3  45.4 


Table  36.  F-15  Baseline  Scenario  Information  for  D-Level  (FY93$-Miliions) 


To  develop  an  Opdon-II  scenario,  the  authors  needed  to  identify  F-15  ATS  testing 
requirement  sets  that  might  be  grouped,  and  thereby,  addressed  by  one  or  more  families  of 
testers.  Five  of  the  F-15  A/B  tester  types  were  assumed  to  be  replaced  by  2  sets  of  family 
testers.  This  was  based  on  two  observations:  (1)  the  Air  Force  plans  to  replace  testing  capa¬ 
bilities  of  the  AN/GSM  229,  230,  231,  232,  and  233  with  the  DST,  and  (2)  the  METS 
replaced  AN/GSM  229,  230  and  231  testers.  That  left  two  other  ATS  family  groupings: 
antenna  and  electronic  warfare  (EW)  testing.  Hie  final  ATS  family  candidate  was  a  down¬ 
sized  tester  to  meet  the  operational  requirements  that  led  the  Air  Force  to  consider  the 
METS  and  DST.  Considering  the  level  of  technology  available  at  the  beginning  of  the  F- 1 5 
life  cycle,  5  sets  of  ATS  families  were  selected  for  this  scenario.  The  scenario  applied  ATS 
mid-life  additions  (updates)  to  accommodate  new  F-15  C/D  testing  needs  for  the  MTS, 
MSIP  antenna,  and  the  TISS.  The  requirements  for  the  METS  were  assumed  to  have  been 
accommodated  by  the  introduction  of  a  DST  family  earlier. 

Finally,  the  authors  noted  that  the  general  purpose  ATS  candidates  would  meet  a 
range  of  depot  testing  needs,  yet  other  depot  testing  needs  were  probably  very  specialized. 
Therefore,  the  ALM-206’s  and  ALM-205’s  were  assumed  to  be  accommodated  by  several 
of  the  candidate  I-lcvel  ATS  families,  and  the  rest  would  be  unique  as  before.  Figure  15 
illustrates  the  F-15  ATS  family  (Option-II)  scenario  used  to  evaluate  the  acquisition  costs. 

The  technology  risk  factor  (discussed  earlier  in  Section  5. 1.1. 3.1,  subparagraph  c.) 
was  also  applied  to  the  tester  quantities  similar  to  the  earlier  scenarios.  Since  a  very  con¬ 
servative  scenario  approach  (that  resulted  in  identifying  five  families  over  the  life  cycle  and 
added  to  this  the  technology  risk  factor)  was  adopted,  a  more  appropriate  prorated  share  of 
sunk  costs  of  $60  million  was  assumed.  As  indicated  in  Table  37,  the  ATS  family  option 
scenario  reduced  the  acquisition  costs  by  $898  million  or  a  35%  saving.  If  the  more  con¬ 
servative  approach  that  applied  the  prorated  share  of  $60  million  for  5  testers  were  used, 
the  results  would  still  have  produced  a  26%  savings. 

The  single  application  of  the  prorated  share  of  development  costs  ($60  million)  was 
further  justified  by  the  fact  that  this  cost  exceeded  the  total  $51.6  million  development 
expenditures  required  for  all  seven  of  the  F-15  A/B  weapon  system  intermediate  level 
testers  at  the  start  of  the  program. 


Table  37.  F-15  Case  Study  Summary  (FY93$-Millions) 


EXPENSE 

CATEGORY 

OPnON-I 

(No 

Change) 

OPTION-II 

(Common 
ATS  Family) 

OPTION-m 

(Common 
ATS  Specs) 

OPTION-IV 

(Common  SW  Environment) 
(Near  Term)  (Long  Term) 

ATE  Development'A/B 

51.6 

35.6 

45.0 

51.6 

51.6 

ATE  Pnxluction-A/B 

485.5 

185.3 

449.1 

485.5 

485.5 

TPS  Development-A/B 

277.4 

146.6 

250.5 

2302 

84.6 

TPS  Production-A/B 

125.1 

82.9 

115.7 

125.1 

125.1 

SUBTOTAL 

939.6 

450.4 

860J 

892.4 

7462 

ATE  Mid*Life  Addi¬ 
tion 

C/D 

408.7 

246.8 

377.0 

408.7 

408.7 

E 

45.4 

31.7 

41.4 

45.4 

45.4 

DST 

86.9 

81.1 

80.2 

86.9 

86.9 

TPS  (New  Dev  &  Prod) 

C/D 

195.8 

219.4 

179.2 

180.9 

135.1 

E 

14.8 

10.8 

13.7 

14.7 

14.3 

DST 

127.7 

112.9 

116.3 

113.7 

70.5 

SUBTOTAL 

879.3 

702.7 

807.8 

8502 

760.9 

A/B  C/D,  E  Dqwt 

ATE  Deve) 

88.1 

88.3 

76.9 

88.1 

88.1 

ATE  Prod 

293.7 

71.9 

271.7 

293.7 

293.7 

TPS  Devel 

255.8 

128.4 

231.0 

212.3 

78.0 

TPS  Prod 

44.6 

100.8 

41.3 

44.6 

44.6 

SUBTOTAL 

682J 

389.4 

6^.9 

638.7 

Prorated  Share  of  ATS 
Family 

0.0 

60.0 

0.0 

0.0 

0.0 

TOTAL 

2,501.1 

1,602.5 

231.4 

2,012.1 

Figure  15.  Option  n,  ATS  Family  Scoiario  for  the  F-15  Weapon  System 


5.1.1.3J  Abrams  (M-1)  Scenario 

The  Abrams  Tank  designated  as  the  M-1  weapon  system  is  already  supported  with 
families  of  ATS:  the  STE  and  DSESTS  in  the  field  and  the  USM-410  in  the  depot.  The 
Army  plans  to  replace  and  augment  the  STE  fault  detection  and  diagnostic  capabilities  with 
the  contact  test  set  (CTS)  plus  the  electro-optical  assembly  (EOA).  The  Army  has  identified 
plans  to  either  update  or  replace  the  aging  DSESTS.  For  Option  1  (No  Change),  Option  III 
(Common  ATS  Specs),  and  Option  IV  (Common  SW  Environment)  the  authors  assumed  a 
mid-life  update  to  the  DSESTS.  Figure  16  illustrates  the  quantities  and  time  frame  that  the 
M-1  ATS  were  introduced  for  these  acquisition  strategy  scenarios.. 


Figure  16.  M>1  Weapon  System  ATS  Evaluation  Scenario  for  Options  I,  HI,  and  IV 

Since  the  M- 1  weapon  system  was  already  using  ATS  families,  the  Option-II  (Com¬ 
mon  ATS  Family)  scenario  needed  to  be  approached  from  a  slightly  different  perspective. 
For  this  scenario,  the  full  IFTE  family  of  testers  (Contact  Test  Set,  Base  Shop  Test  Facility 
(BSTF),  and  Commercial  Equivalent  Equipment  (CEE))  were  assumed  to  be  available  at 
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the  onset  of  the  M-1  program.  The  Army’s  projected  support  capabilities  of  integrated  sets 
of  CTS,  BSIP  and  CEE  were  used  as  the  starting  point: 

•  2  depot  USM-410  could  be  replaced  by  2  CEE, 

•  198  DSESTS  could  be  replace  with  53  BSTF, 

•  737  STE  could  be  replaced  by  894  CTS  augmented  with  86  EGAs. 

To  back  into  the  number  of  ATS  required  at  the  beginning  of  the  program,  the  tech¬ 
nology  risk  factor  was  applied  to  the  BSTF  quantity  (e.g.,  same  approach  used  on  the  other 
case  studies). 

The  CTS  presented  a  problem,  since  it  is  not  a  direct  1-for-l  replacement  of  the 
STE.  The  CTS  is  projected  to  reduce  the  woric  load  relative  to  the  current  STE  while  pro¬ 
viding  increased  fault  isolation  ctq)abilides.  At  the  same  time  technology  differences  ten 
years  earlier  would  have  posed  difficulties  in  fielding  a  CTS  with  comparable  performance. 
Therefore,  for  this  scenario  case  study,  the  CTS  quantity  was  assumed  to  remain  constant. 

Finally,  25%  of  the  technology  (or  resting  requirements)  were  assumed  to  change 
at  the  mid-life  update,  and  the  same  approach  used  on  the  other  case  studies  was  applied. 
Figure  17  illustrates  die  M-1  ATS  family  scenario  used  to  evaluate  the  acquisition  costs. 


Figure  17.  Option  n,  ATS  Family  Scenario  for  M-1  Weapon  System 


The  baseline  scenario  information  used  to  develop  both  figures  is  presented  in  Ta¬ 


ble  38. 


Table  38.  M-1  Baseline  Scraario  Information 


ATSNmm 

QTY 

Card! 

*  *  - 

MiMMCJ 

ATE 

Devet 

($M) 

ATE 

Acq 

($M) 

TPS 

Devd 

(SM) 

TPS 

Acq 

<$M) 

No. 

ITAs/ 

AIS 

AN/USM-<10 

2. 

Yes 

Yes 

0.7 

3.2 

6.8 

1.4 

34 

DSESTS 

198 

Yes 

Yes 

8.4 

179.3 

5.1 

30.3 

34 

STEMl/FVS 

737 

No 

Yes 

0.4 

293 

0.4 

22.5 

1 

CTS 

896 

Yes 

Yes 

0.0 

10.1 

1.0 

0.1 

1 

EOA(CTS) 

86 

Yes 

Yes 

2.1 

37.1 

1.0 

3.4 

1 

BSTF 

53 

Yes 

Yes 

14J 

116.6 

5.4 

53 

21 

CEE 

2 

Yes 

Yes 

0.5 

3.7 

3.9 

03 

15 

The  results  of  the  M-1  case  study  scenario  are  presented  in  Table  39.  Although  Op¬ 
tion  n,  Common  ATS  Families,  has  the  lowest  relative  total  cost,  the  total  cost  differences 
were  not  as  high  as  computed  in  the  other  case  studies.  One  subjective  rationale  for  this 
condition  is  that  some  of  the  ATS  being  replaced  by  this  scenario  are  aging  ATS  that  had 
themselves  been  part  of  a  family  group  used  by  the  Army. 


Table  39.  M<1  Case  Study  Summary  (FY93$-Million) 


EXPENSE 

CATEGORY 

opnoN-i 

(No 

Change) 

opnoN-n 

(Commtxi 

ATSFamOy) 

OPTION-m 
(Common 
ATS  Spec) 

OPnoN-iv 

(Common  SW  Environment) 
(Near  Term)  (Long  Term) 

ATE  Development 

9.4 

4.6 

8.2 

9.4 

9.4 

ATE  Production 

211.8 

177.9 

196.0 

211.8 

211.8 

TPS  Development 

12.4 

10.2 

11.2 

10.3 

3.8 

TPS  Production 

54.2 

9.8 

50.1 

54.2 

54.2 

SUBTOTAL 

287,8 

ms 

265,5 

285.7 

2192 

ATE  Mid  Life  Addition 

49.2 

0.0 

45.4 

49.2 

49.2 

TPS  (New  Dev  &  Prod) 

5.6 

0.0 

5.1 

5.2 

4.1 

SUBTOTAL 

54,8 

0,0 

50,5 

54.4 

53J 

ATE  Mid  Life  Update 

47.9 

33.7 

44.2 

47.9 

47.9 

TPS  Rehost 

3.9 

3.7 

3.9 

2.4 

2.4 

Table  39.  M-1  Case  Study  Summary  (FY93$-Million)  (Continued) 


EXPENSE 

CATEGORY 

OPTION-I 

(No 

Change) 

OPTION-n 

^Common 
ATS  Family) 

opnoN-m 

(Common 
ATS  Spec) 

opnoN-iv 

(Common  SW  Environment) 
(Near  Term)  (Long  Term) 

SUBTOTAL 

51A 

37.4 

48.1 

SOJ 

50J 

Prorated  Share  of  ATS 
Family 

0.0 

60.0 

0.0 

0.0 

0.0 

TOTAL 

394.4 

299.9 

364.1 

390.4 

382A 

5.1.U.6  Sensitivity  Analysis 

A  separate  analysis  was  performed  to  assess  the  sensitivity  of  the  case  study  results 
to  variations  of  selected  boundary  condition  assumptions.  This  sensitivity  analysis  revealed 
that  the  ATS  development  and  production  costs  were  relatively  insensitive  to  large  changes 
in  (1)  the  option  cost  factors,  (2)  technology  adjustment  factor,  and  (3)  mid-life  ATS  con¬ 
figuration  changes.  The  major  cost  savings  for  ATS  family  acquisition  were  attributed  to 
the  reduction  of  duplicative  engineering  tasks  and  production  economies  of  scale  for  com¬ 
mon  equipment 

A  sensitivity  analysis  approach  was  adopted  that  computed  the  p^cent  change  from 
the  nominal  conditions  for  various  parameter  and  option  cost  factor  changes.  Selected  case 
parameters  in  the  case  study  model  were  varied  and  the  change  in  estimated  acquisition 
costs  computed. 

Table  40  presents  the  results  of  a  comparison  for  the  F- 1 8  Option-n  (Common  ATS 
Family)  scenario.  Percentage  changes  are  presented  from  two  po-spectives:  relative  to  the 
Option-n  scenario  (e.g.,  factors  and  parameters  identical  to  those  used  in  the  F- 18  (Tase 
Study),  and  relative  to  the  baseline  (No  Change)  Option-I  scenario.  These  two  scenario  re¬ 
sults  are  sunrunarized  above  the  bold  line  in  this  table. 

Clearly,  based  on  this  data,  tiie  model  was  relatively  insensitive  to  these  parameter 
changes.  The  sources  of  greatest  savings  in  each  of  the  case  studies  were  (1)  the  cost  avoid¬ 
ance  of  duplicative  non-recurring  engineering  development  of  both  ATE  and  TPSs,  (2)  re¬ 
duced  ATE  quantities  attributed  to  general  purpose  ATS  family  candidates  that  meet 
multiple  applications,  and  (3)  production  economies  of  scale. 

An  analysis  was  also  conducted  to  determine  how  well  the  technology  adjustment 
factor  compensated  for  technology  that  was  not  available  at  the  onset  of  a  program.  Since 
the  F-16  A/B  AIS  was  available  at  the  onset  of  the  F-18  case  study  scenario,  the  cost  (or 
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quantity)  did  not  need  to  be  adjusted  by  the  25%  technology  adjustment  factor.  Therefore, 
the  sensitivity  analysis  was  based  on  the  F-18  Option-II  scenario  using  the  F- 16  A/B  AIS 
as  the  candidate  family  tester.  This  approach  would  help  discount  an  unknown  variable, 
specifically  the  technology  adjustment  factor. 

The  results  revealed  that  the  total  acquisition  costs  varied  by  as  much  as  $150  mil¬ 
lion  on  either  side  of  the  original  confuted  value  that  had  applied  the  25%  technology  ad¬ 
justment  factor  to  the  CASS  quantities.  This  sensitivity  analysis  revealed  that  the  difference 
in  acquisition  cost  was  less  than  11%  relative  to  the  baseline  for  a  range  of  assumptions 
(from  very  conservative  to  very  extreme).  The  wide-range  of  assumptions  that  produced 
this  relatively  snudl  percentage  included  work  loading  of  a  single  RF  CASS  station  versus 
a  full  F- 16  AIS,  introduction  of  card  level  testing  capabilities,  and  the  addition  of  F- 1 6  AIS 
equivalent  P^I  sustaining  engineering  costs. 


TiMe  40.  F<18  Option-II  Sc^ario  Soisitivity  Analysis 


CASE  PARAMETER 

Acquisition 

Cost 

(FY93$-M) 

%  Change 
Relative  to 
Option-II 
Scoiario 

%  Change 
Relative  to 
Baseline 
Scenario 

Baseline  (No  Oiange)  Scenario  Cost 

1320.5 

0 

N/A 

Option-D  (ATS  Family)  Scenario  Cost 

576.5 

N/A 

0 

ATE  Development  Option  Cost  Factor  •  Increase 
from  0.05  io  0.10 

594.5 

3.0 

1.3 

ATE  Acquisition  Option  Cost  Factor  (Learning 
Curve)  -  Increase  from  0.85  to  0.90 
-  Increase  from  0.85  to  0.95 

588.6 

600.6 

2.1 

4.0 

0.9 

1.8 

TPS  Acquisition  Option  Cost  Factor  (Learning 
Curve)  -  Increase  from  0.85  to  0.95 

591.5 

2.5 

1.1 

TPS  Re-Host  Option  Cost  Factor  -  Increase  from 
0.33  to  1.0 

642.1 
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5.0 

Mid-Life  ATS  Configuration  Change  •  Increase 
from  25%  to  50% 

5992 

3.8 

1.7 

Technology  Adjustment  Factor  (Compensate  for 
Eariier  Insertion)  -  Increase  from  25%  to  50% 

602.8 

4.4 

2.0 

The  authors  were  concerned  diat  many  of  its  assumptions  might  have  been  too  con¬ 
servative,  and  the  range  of  estimated  savings  were  too  low.  Figure  18  illustrates  the  range 
of  potential  savings  for  the  F- 16  ATS  under  (Dption-ll,  if  a  more  aggressive  set  of  assump- 


tions  had  hoea  adopted.  The  top  line  of  numbos  reflect  the  results  of  the  F-16  case  study 


presented  earlier. 

Option-I 

Option-II 

(Baseline  Data) 

(ATS  Family) 

ATS  Devel  & 
Prod  Costs 

$2374M 

^  $1716M  Conservative  Total 

V  $1343M  Adjusted  Total 


Range  of  Potential 
Savings 

$658M  (27%) 
to 

$1031M  (43%) 


Station  Work  Load 
*2®^“  Adjustment 

.  iHAM  as  many 

Systems  Using 

1  AVI  Reflect  less  Com- 
plex  Depot  ITAs 


Note:  Additional  cost  savings  could  have  been  realized  in  this  sce¬ 
nario  if  ATS  families  also  supported  other  electronic  testing 
requirements  such  as  EW  and  engine  controllers. 


Figure  18.  Additional  F-16  ATS  Savings  Potential  for  an  Option-II  Acquisition 

Strategy 


The  $203  million  reduction  represents  a  station  workload  adjustment  As  stated 
briefly  in  the  case  study,  a  more  appropriate  station  workloading  would  be  2  or  3  CASS  RF 
stations  to  1  full-up  F-16  AIS  consisting  of  4  station  types.  Although  the  4  F-16  AIS  station 
types  were  appropriately  workloaded  at  the  onset  of  the  F-16  program,  avionics  subsystem 
reliability  improvements,  such  as  that  for  the  inertial  navigation  system  (INS),  have  signif¬ 
icantly  changed  the  station  workloading  of  die  F-16  AISs.  Had  die  F-16  AISs  been  more 
general  purpose  in  nature,  as  opposed  to  subsystem  specific,  this  difference  might  not  be  as 
pronounced.  ATS  quantiQr  reductions  are  reflected  in  this  estimated  additional  $203  mil¬ 
lion  savings. 

The  $60  million  reduction  assumes  that  twice  as  many  systems  will  eventually  use 
the  ATS  family,  thereby,  effectively  distributing  the  ATS  development  costs  across  more 
system  applications.  Finally,  the  $110  million  is  based  on  the  assumption  that  the  depot 
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ATS  ITAs  would  not  be  substantially  more  con^lex  than  those  currently  used  in  the  depots. 
The  approach  used  in  the  F-16  case  study  estimated  the  ITA  costs  based  on  I-level  devel¬ 
opment  and  production  costs.  This  appears  to  be  very  high,  but  comparable  depot  data  was 
not  available  to  develop  a  more  representative  costing  model.  From  this  perspective,  the  au¬ 
thors  observed  that  the  ATS  Family  acquisition  strategy  had  the  potential  of  producing  an 
additional  16%  savings  over  and  above  that  estimated  in  this  case  study. 

Alternatively,  a  case  may  be  made  that  dte  F-16  ATS  acquisition  strategy,  which 
was  concurrent  with  weapon  system  development  and  ongoing  support,  was  driven  to  some 
degree  by  operational  support  requirements  and  foreign  military  sales  commitments.  An  es¬ 
timate  of  savings  potential  of  an  Option  II  scenario  based  on  the  F-16  ATS  cost,  less  the 
identified  P^I  sustaining  engineering  costs,  was  still  significant  and  computed  to  ranged 
from  15%  to  34%  using  this  same  approach. 

A  final  evaluation  addressed  the  influences  of  (1 )  using  actual  data  for  the  RF  CASS 
or  (2)  applying  the  computer  factors  against  the  baseline  (no  change)  Opdon-I  scenario 
data.  Several  alternate  Option-II  sensitivity  analyses  were  run  in  which  the  option  cost  fac¬ 
tors  were  tq)plied  directly  to  baseline  data.  In  the  case  of  the  F-16,  directly  applying  the  fac¬ 
tors  to  the  baseline  data  resulted  in  a  total  t^uisition  cost  that  was  approximately  $44 
million  less  than  the  original  value  that  was  computed  using  the  actual  CASS  RF  data. 

For  the  F- 18  case  study,  the  alternate  approach  still  showed  a  savings  of  approxi¬ 
mately  $345  million,  however,  it  was  substantially  higher  than  die  original  computed 
tion-n  total  acquisition  cost  Upon  evaluation  of  the  specifrc  data  fields  in  this  analysis,  the 
approach  that  applies  the  Option-II  cost  factors  directly  to  the  baseline  data  was  found  to 
result  in  a  very  conservative  estimate  of  total  costs.  The  main  contributing  reason  for  its 
conservative  nature  was  that  it  does  not  introduce  the  potential  reduction  of  tester  quantities 
that  were  observed  with  the  actual  data. 

Consequently,  using  the  CASS  RF  data  is  more  representative  of  what  would  result 
from  a  new  ATS  acquisition  strategy.  However,  later  in  Section  6,  when  computing  esti¬ 
mated  ATS  investment  savings,  the  more  conservative  approach  is  used  for  model  simpli¬ 
fication. 

5.1,2  Operations  and  Support  Costs 

Operating  and  support  (O&S)  costs  for  each  weapon  system  were  difficult  to  obtain 
since  the  Services  do  not  collect  and  accumulate  detailed  support  costs  for  automatic  test 
systems.  The  analysis  presented  in  this  section  may  be  used  to  indicate  trends;  however  it 
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should  not  be  considered  conclusive  due  to  the  limited  O&S  data  available.  The  main 
sources  of  O&S  cost  information  included  an  evaluation  of  the  IFTE  support  for  the  AH- 
64  Apache  helicopter  (APACHE  1991],  the  IFTE  Cost  and  Operational  Effectiveness 
Analysis  (COEA)  [IFTE  1 989],  data  from  the  CASS  training  and  manpower  analysis  which 
was  used  to  develop  the  1991  CASS  Navy  Training  Plan  (NTP),  and  several  programmatic 
CASS  briefings  [NAJRIAN  1991]  [EVANOVICH]. 

The  study  team  collected  available  cost  data  for  six  different  O&S  cost  categories: 
Operator  and  Maintainer  Training,  Support  of  ATS,  Supply  Support,  Technical  Data,  Man¬ 
power,  and  Facilities.  Most  of  the  collected  data  in  these  categories  may  be  best  described 
as  anecdotal  information,  yet  these  data  also  provide  evidence  of  trends.  The  most  consis¬ 
tent  trend  observed  indicated  that  weapon  system  support  is  more  effective  with  a  conunon, 
as  opposed  to  weapon  system  unique,  ATS.  A  common  ATS  family  limits  the  amount  of 
support  needed  for  a  single  system  while  simplifying  other  aspects  of  weapon  system  sup¬ 
port. 

The  following  discussions  of  O&S  costs  are  broken  down  into  five  areas:  Technical 
Publications;  Support  of  ATS;  Manpower,  Personnel  and  Training;  Supply  Support;  and 
Facilities. 

S.U.1  Technical  Publications 

The  primary  source  of  infomution  on  technical  publications  was  obtained  fi-om  the 
Naval  Aviation  Technical  Services  Facility  (NATSF)  on  CASS  transition  programs.  For  5 
initial  ATS  systems  to  be  transitioned  there  are  a  total  of  367  technical  manuals.  NATSF 
estimated  that  approximately  $100,000  is  spent  annually  (by  NATSF  and  other  Navy  field 
activities)  to  keep  these  existing  ATS  manuals  current  These  resources  are  necessary  pri¬ 
marily  to  maintain  current  information  due  to  obsolescence-driven  ATS  spare  parts  chang¬ 
es  and  continuing  engineering  changes  needed  to  meet  evolving  test  requirements.  NATSF 
representatives  indicated  this  is  a  conservative  estimate,  and  the  costs  could  run  higher  de¬ 
pending  on  the  number  of  changes  needed  to  maintain  an  ATS  configuration. 

In  contrast,  CASS  requires  only  13  technical  manuals  for  its  four  configurations. 
The  CASS  use  a  electronic  technical  manuals  system  on  optical  disks,  and  each  CASS  sta¬ 
tion  has  the  capability  to  display  and  print  its  own  technical  documentation.  Updates  of 
technical  manual  information  are  to  be  accomplished  by  distributing  an  optical  disk  rather 
then  by  the  traditional  use  of  change  pages.  Although  the  Navy  has  not  estimated  the  total 
savings,  they  indicated  an  overall  technical  publications  cost  reduction  is  anticipated.  An 


additional  benefit  involves  reduced  facilities  requirements  attributed  to  reduced  volumes  of 
paper. 

5.UJ  Support  of  ATS 

Based  on  discussions  with  Navy  field  activity  support  personnel,  the  authors  noted 
that  between  80-200  line  items  of  support-of-support  (SOS)  support  equipment  are  re¬ 
quired  to  support  any  particular  ATE.  These  items  include  maintenance  equipment  such  as 
osciUoscopes  and  multimeters;  support  software  such  as  self-maintenance  test  program  sets 
and  their  documentation;  special  tools  such  as  card  extractors,  pin  insertion  and  removal 
tools;  and  unique  calibration  equipment  In  contrast  CASS,  needs  approximately  135  items 
of  support  equipment  across  its  4  configurations.  Potential  savings  are  expected  due  to  re¬ 
ductions  of  SOS  support  equipment  sets.  For  example,  CASS  requires  one  set  whereas  six 
unique  sets  are  required  for  the  6  multiple  system  peculiar  ATE  that  will  be  replaced  by  the 
planned  off-load:  EOSTS,  HATS,  VAST,  ESTS,  TMV  and  NEWTS.  In  addition,  other  sys¬ 
tem-unique  ATS  savings  are  anticipated  due  to  elimination  of  multiple  sets  of  ATS  suppon 
equipment  Other  potential  support  equipment  (for  ATS)  cost  saving  categories  include; 
training,  maintenance  personnel,  technical  publications,  and  support  equipment  storage 
space. 

5.1,2  J  Manpower,  Personnel,  and  Training 

Manpower,  personnel,  and  training  savings  attributed  directly  to  standard  ATS  fam¬ 
ilies  have  been  identified  by  bodi  the  Army  and  Navy.  The  following  paragraphs  present 
the  identiBed  savings  for  C^SS  and  IFTE  ATS  families. 

The  1991  CASS  Transition  Plan  (information  provided  by  the  Navy  in  the  TIMs) 
was  used  as  a  basis  for  assessing  personnel  and  training  requirements  for  the  6  initial  testers 
being  replaced  by  CASS;  EOSTS,  HATS,  VAST,  ESTS,  TMV  and  NEWTS.  The  Navy  es¬ 
timated  that  871  people  Navy-wide  are  needed  to  currently  operate  and  support  these  6 
testers.  Annual  support  training  requirements  for  operators  and  maintainers  associated  with 
these  6  testers  are  budgeted  at  $2,514  million.  In  contrast,  about  610  equivalent  people  are 
required  to  man  the  CASS  stations.  The  equivalent  training  cost  associated  with  the  CASS 
is  estimated  to  be  $0,579  million  This  represents  a  porsormel  reduction  of  about  40%,  and 
a  training  cost  reduction  of  greater  than  75%.  The  training  savings  are  based  on  a  reduction 
of  training  courses  from  22  for  the  6  testers  to  3  for  CASS.  Tables  41  and  42  show  both  the 
current  training  requirements  for  the  6  testers  and  equivalent  training  requirements  for 
CASS. 
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Tible  41.  Current  ATE  IVaining  Requirements  and  Costs 


SYSTEM 

NEC/MOS' 

BILLETS 
(TOTAL  NAVY) 

ANNUAL 
TRAINING 
COSTS  (FY93$) 

AN/AAM-60 

6684/ 

41 

0.23  IM 

(HOSTS) 

7959 

67 

AN/USM-403 

6619/ 

37 

0.1 38M 

(HATS) 

6628 

39 

ANAJSM-247 

6652/ 

116 

0.987M 

(VAST) 

6653/ 

132 

6665/ 

52 

6659 

71 

AN/ASM-614 

6608 

86 

0.090M 

(ESTS) 

AN/USM-470 

6694/ 

45 

0.1 26M 

(V)2  (TMV) 

6695 

20 

AN/USM-458 

6618/ 

114 

0.942M 

(NEWTS) 

6482/ 

51 

6622 

Summary  for  6 

14NEC/MOSS 

871  PEOPLE 

$2.514M 

Systems 

(1)  Navy  Enlisted  Qassibcation/Military  Occupational  Speciality 

Table  42.  CASS  Equivalent  Training  Requirements  and  Costs 


SYSTEM 

NEC/MOS 

BILLETS 
(TOTAL  NAVY) 

ANNUAL 
TRAINING 
COSTS  (FY93$) 

CASS 

Sununaiy  for  1 
System 

6704/6705 

(2  NEC/MOSs) 

610 

PEOPLE 

$0.579M 
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Another  example  of  personnel  and  training  savings  is  illustrated  by  the  Army's 
IFTE  ATS.  The  Army  IFTE  Cost  and  Operational  Effectiveness  Analysis  (COE A)  evalu¬ 
ated  IFTE  against  12  system-specific  ATE  systems  [IFTE  1989].  The  IFTE  case  was  based 
on  a  Europe  VI  scenario  (an  Army  operational  scenario  based  in  Europe)  and  included  per¬ 
sonnel  from  a  corps  and  echelon  above  core  slice.  The  training  cost  was  an  estimate  based 
on  total  annual  operator  and  maintainer  personnel  training  requirements  for  the  12  system- 
specific  testers  versus  total  annual  operator  and  ntaintainer  personnel  training  requirements 
for  the  IFTE.  The  personnel  requirements  at  a  three-level  maintenance  strategy  were  esti¬ 
mated  to  be  641  people  with  an  annual  training  cost  of  $4.8  million  for  the  system-specific 
ATS.  This  was  compared  to  269  personnel  required  to  operate  and  maintain  the  IFTE  ATS 
with  an  annual  training  cost  of  $2.5  million  performing  the  same  functions  as  the  12  sys¬ 
tem-specific  ATSs.  This  represents  a  personnel  savings  of  about  42%  and  a  training  cost 
savings  of  52%. 

5.1^.4  Supply  Support 

Navy  projections  indicated  the  total  number  of  ATE  related  spare  part  line  items 
could  be  reduced  from  30,000  to  3,800  per  aircraft  carrier  by  replacing  existing  ATE  with 
CASS.  Savings,  although  not  quantifiable  in  this  study,  are  associated  with  line  items  that 
need  to  be  managed,  tracked,  stocked,  account  for,  and  kept  in  a  “ready  for  issue”  con¬ 
dition.  A  standard  ATS  family  will  reduce  the  total  number  of  SOS  line  items  over  system 
specific  ATS  by  virtue  of  common  assets  for  a  much  smaller  number  of  ATS  configurations 
[MERRILL]. 


5.1,2.5  Facilities 

The  Navy  indicated  facility  space  requirements  could  be  reduced  by  approximately 
5,000  square  feet  on  a  typical  aircraft  carrier  by  replacing  the  existing  ATS  with  CASS. 
Their  studies  project  total  facility  reductions  from  15,000  square  feet  to  10,(X)0  square  feet 
floor  space.  This  reduction  in  facility  space  requirements  includes  space  reductions  for  sup¬ 
port  equipment,  spare  and  repair  parts,  personnel,  and  technical  manuals  [MERRILL]. 

5.2  QUALITATIVE  ANALYSIS  RESULTS 

The  results  of  the  qualitative  assessment  were  inconclusive.  Even  though  the  qual¬ 
itative  results  were  inconclusive,  the  approach  helped  bring  out  several  subjective  observa¬ 
tions  by  team  members  working  on  this  portion  of  the  analysis.  Therefore,  the  following 
paragraphs  are  intended  to  provide  insight  into  their  thought  process  and  summarize  the  ob- 
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sovations  upon  which  they  gained  a  consensus.  The  observations  covered  three  areas;  (1) 
each  of  the  strategy  options  would  be  able  to  accommodate  technology  and  test  require¬ 
ments  growth  from  within  its  respective  framework;  (2)a  common  ATS  family  strategy  was 
the  principal  option  that  would  permit  operational  commonality,  but  other  resources  (TPSs, 
ITAs,  etc.)  would  be  needed  also  in  addition  to  standard  ATS;  and  (3)  depot  efficiency  did 
not  appear  to  favor  any  one  of  the  option  strategies  over  another. 

The  approach  attempted  to  use  four  evaluation  criteria  for  assessing  the  individual 
strategies:  operational  commonality,  technological  feasibility,  programmatic  risk,  and  de¬ 
pot  efficiency.  These  four  criteria  categories  were  then  broken  down  into  subcategories  and 
evaluated.  Figure  19  illustrates  the  criteria  and  subcategories  used  in  the  evaluation.  Ser¬ 
vice  representatives  helped  to  establish  the  subcriteria  categories.  The  following  are  the  cat¬ 
egory  assessment  criteria  used  in  this  analysis 

a.  Operational  Commonality:  Assessment  of  the  opportunities  for  and  the  limita¬ 
tions  of  increased  inter-system  and  intra-Service  and  inter-Service  application  of 
ATS. 

b.  Technological  Feasibility:  Assessment  of  option  evolution  opportunities  and 
limitations  in  terms  of  obsolescence,  technology  growth,  and  test  requirements 
growth. 

c.  Programmatic  Risks  and  Opportunities:  Assessment  of  option  compatibility  with 
ATS  requirements  derived  from  mission  needs,  maintenance  concepts,  testing 
requirements,  and  weapon  system  and  ATS  acquisition  strategies. 

d.  Depot  Efficiency:  Assessment  of  ATS-related  factors  influencing  depot  effi¬ 
ciency  and  the  impact  on  both  depot  competition  and  Service  operational  main¬ 
tenance  requirements  and  capabilities. 

A  pair-wise  evaluation  concept  was  used  to  compare  the  strategies.  A  commercial 
software  tool  that  runs  on  the  standard  personal  computer  was  used  to  perform  the  evalua¬ 
tion.  The  concept  implemented  by  this  tool  required  that  each  of  the  subcategories  be  eval¬ 
uated  to  determine  its  level  of  importance  relative  to  each  of  the  other  categories.  This  was 
accomplished  by  comparing  pairs  of  subcategories.  The  categories  were  than  ranked 
against  each  of  the  study  options  to  determine  what  efrect  they  had  on  the  options.  Study 
team  members  (including  people  from  OSD  and  each  of  the  Services)  were  asked  to  per¬ 
form  the  category  ranking.  This  approach  was  used  in  an  attempt  to  quantify  this  very  sub¬ 
jective  topic  area.  We  observed  that  perceptions  of  individuals  participating  in  the  ranking 
process  could  significantly  alter  the  analysis  results.  Therefore,  the  results  of  pair-wise 


Improve  ATS 
Acquisition 


1  ■ 

Operational  1 
Conunonaliiy  1 

Technological  1 
Feasibility  1 

Programmatic  Risks! 
and  Opportunities  1 

DqxM  1 

Efficiency  1 

Service  Infiastnicture 

Field  Logistics  Tail 

Fkxil^ty  to  Meet 
Miraion 

Mssion  Suitability 

Inier/Intra  Servicing 

Flexibility  to  Sigtport 
Changing  Test  Req 

Badcwvds 

Conqaiibility 

Flexil^tylD 
Accommodate  ^uipment 
Obsolesoence 

Ability  to  Meet  W/S 
Cost  Objectives 

Ability  to  Meet  Program 
_  Schedule 

Ability  to  Meet  Technical 
__  Requirements 

Ability  to  fteet  Mission 
Needs 

Benefits  of  Centralized/ 
Decentralized  ATS 

DqxM  Intefservidng 

Increased  Public/Private 
DqxM  Competition 

Ability  to  Meet  Depot 
Technical  Requirements 

Figure  19.  ATS  Acquisition  Strategy  Qualitative  Analysis  Evaluation  Criteria 


comparison  analysis  were  suspect  However,  the  participants  felt  that  tfie  consensus  estab- 
Uriied  on  three  areas  during  subjective  discussions  was  extremely  valuable.  The  following 
paragraphs  highlight  the  consoisus  established. 

5.2.1  Operational  Ccunnionality 

Several  factors  were  identified  diat  influence  opportunities  for  improving  opera¬ 
tional  conunonality  (both  cross-weapon  support  as  well  as  cross-Service  support).  These 
factors  are  Service  infrastructure,  field  logistics  tail,  flexibility  to  meet  mission  require¬ 
ments,  mission  suitability,  and  the  ease  at  which  inter  and  intra  Servicing  can  be  achieved. 
Each  of  these  factors  was  evaluated  to  determine  which  option  would  support  maximum 
operational  conunonality. 

During  this  study,  the  study  team  learned  that  the  Air  Force  had  requested  avionics 
repair  support  from  Navy  aircraft  carriers  in  the  Gulf  during  Desert  Storm.  This  repair  sup- 
p<wt  did  not  occur  since  the  Air  Force  avionics  and  support  requirements  differed  signifi- 
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cantly  from  the  capabilities  available  with  the  Navy  forces.  However,  if  appropriately 
planned,  ATS  compatibility  could  have  allowed  the  Air  Force  to  receive  avionics  repair 
support  from  Navy  carriers.  This  example  influenced  most  of  the  discussions.  Compatible 
ATSs  across  weapon  systems  would  allow  the  flexibility  of  wartime  tester  work-loading, 
help  reduce  unique  tester  deployment  logistics,  and  permit  a  more  flexible  use  of  mainte¬ 
nance  personnel.  Service  members  felt  that  if  there  is  a  trend  towards  more  joint  operations, 
then  there  will  also  be  new  requirements  for  increased  operational  commonality  in  terms  of 
cross-Service  maintenance. 

The  study  team  observed  that  cross-weapon  support  of  this  nature  would  require  ex¬ 
tensive  and  detailed  advanced  support  planning.  The  additional  planning  and  resources 
could  represent  significant  logistics  costs  depending  on  the  specific  systems  involved.  The 
participants  also  felt  that  additional  analysis  in  this  very  specific  cross-Service  topic  area 
was  needed.  However,  there  was  agreement  that  increased  ATS  commonality  would  great¬ 
ly  enhance  the  feasibility  of  pursuing  this  opportunity.  Cross-weapon  and  cross-Service 
support  and  therefore  operational  convnonality  would  be  enhanced  by  the  use  of  identical 
or  compatible  automatic  test  systems.  The  use  of  common  ATS  families  would  allow  flex¬ 
ibility  of  tester  workloading,  and  possibly  reduce  total  ATS  requirements.  This  was  the 
only  option  strategy  that  had  the  potential  to  increase  flexibility  of  weapon  system  support, 
reduce  deployment  resource  requirements,  and  permit  more  flexible  use  of  maintenance 
personnel. 

5,2^  Technological  Feasibility 

For  the  purposes  of  this  evaluation,  technological  feasibility  was  defined  as  the  abil¬ 
ity  of  ATS  hardware  and  software  to  accommodate  changing  weapon  system  test  require¬ 
ments.  In  this  context,  technological  feasibility  is  affected  by  ATS  hardware  and  software 
architecture.  The  flexibility  of  the  architecture  determines  how  quickly  the  ATS  can  ac¬ 
commodate  changes  in  technology  and  testing  requirements.  Several  subcriteria  that  influ¬ 
ence  this  capability  were  identified  such  as  the  flexibility  to  support  changing  testing 
requirements,  backward  con^atibility,  and  the  flexibility  to  compensate  for  equipment  ob¬ 
solescence. 

The  participants  in  this  analysis  agreed  tfiat  each  of  the  options  appeared  able  to  ac¬ 
commodate  technology  growth,  test  requirements  growth,  and  obsolescence  avoidance 
within  its  own  framework.  Each  option  was  able  to  support  changing  test  requirements  by 
accommodating  a  capability  commensurate  widi  die  needed  testing  requirements.  Opinions 
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of  the  team  members  did  differ  slightly  on  the  rate  that  each  option  might  accommodate 
technology  change.  There  was  general  agreement  that  Option  in  (Common  ATS  Specifica¬ 
tions)  had  the  greatest  potential  of  limiting  the  rate  of  technology  growth.  There  was  a  gen¬ 
eral  perception  that  the  ATS  and  technologies  used  by  them  evolved  faster  than 
implemented  standards. 

SJ23  Programmatic  Risks  and  Opportunities 

Several  factors  were  identiHed  that  influence  ATS  programmatic  risks  and  oppor¬ 
tunities.  These  factors  included 

•  the  ability  of  an  ATS  option  to  meet  weapon  system  cost  objectives, 

•  the  ability  of  an  ATS  option  to  meet  weapon  system  schedule  requirements, 

•  the  ability  of  an  ATS  option  to  meet  weapon  system  mission  needs,  and 

•  the  benefits  of  centralized  managed  ATS. 

All  of  the  options  were  assessed  to  be  compatible  with  the  first  four  subcriteria  con¬ 
straints;  however,  centralized  versus  decentralized  management  of  an  ATS  was  perceived 
by  the  Services  as  sensitive  to  the  success  of  any  particular  weapon  system  program.  This 
was  consistent  with  the  i^ing  Aat  ATS  were  managed  differently  within  the  Services. 
Both  ATE  and  TPSs  were  found  to  be  a  mix  of  centralized  and  decentralized  management 
activities  within  the  Services.  Table  43  illustrate  the  management  relationships  observed.- 
During  the  period  of  the  study,  the  Navy  was  in  the  process  of  transitioning  from  ATS  cen¬ 
tralized  activities  for  NAVAIR  to  ATS  centralized  activities  Navy  wide. 


Table  43.  ATE>TPS  Centralized  Decentralized  Management 


ATS  Element 

Air  Force 

Army 

Navy 

ATE 

Decentralized 

Centralized 

Centralized 

TPS 

Eiecentralized 

Decentralized 

Centralized 

The  results  of  this  portion  of  the  analysis  were  inconclusive  as  a  direct  consequence 
of  these  management  approach  differences.  Evaluations  were  split  directly  along  Service 
lines:  (1)  Army  and  Navy  representatives  felt  that  having  a  mature  ATS  ready  to  support 
varying  weapon  system  requirements  tended  to  reduce  weapon  system  risks.  (2)  Air  Force 
representatives  tended  to  feel  that  a  predetermined  ATS  configuration  increased  weapon 
system  risk  since  the  ATS  may  not  meet  all  specific  weapon  system  test  requirements. 
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5^.4  Depot  Efficiency 

Analysis  of  the  depot  efficiency  criteria  did  not  appear  to  favor  any  of  the  options. 
Instead,  the  Service  members  subjectively  observed  that  selected  aspects  of  each  of  the  op¬ 
tions  had  the  potential  of  improving  depot  efficiency.  For  example,  Service  representatives 
felt  that  improved  access  and  delivery  of  product  data  along  with  standardized  COTS  ATE 
would  tend  to  increase  their  efficiency,  reduce  cost  and  help  foster  depot  competition.  They 
also  acknowledged  that  standard  instrument  interfaces,  especially  if  widely  used  in  the 
commercial  sector,  could  improve  depot  efficiency. 
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6.  DOD  ATS  INVESTMENT  ANALYSIS 


The  objectives  of  this  analysis  were  to  characterize  total  DoD  ATS  investments  and 
to  project  potential  ATS  acquisition  savings  based  on  a  new  DoD  ATS  investment  strategy. 
The  analysis  results  revealed  that  DoD’s  ATS  inventory  is  around  $45-50  billion,  and 
projected  that  DoD  ATS  acquisition  expenditures  across  FY93  through  FY99  will  total  an 
additional  $11  billion  if  no  acquisition  strategy  changes  are  adopted.  The  results  also 
projected  a  potential  net  ATS  investment  strategy  saving  of  over  $  1 .5  billion  over  this  same 
poiod  if  a  new  DoD  ATS  investment  strategy  is  adopted. 

This  analysis  makes  extensive  use  of  ATS  baseline  data  and  investment  strategy 
option  information  presented  earlier.  Past  and  projected  DoD  weapon  systems  acquisition 
budget  information  is  used  throughout  the  analysis.  The  approach 

•  uses  the  ATS  acquisition  ratios  that  were  grouped  by  weapon  system  commodi¬ 
ties,  and 

•  iqipUes  these  ratios  to  past  and  projected  weapon  system  expenditures. 

A  model  of  ATS  expenditures  was  created  by  combining  the  results  of  these  two 
steps.  The  data  used  to  develop  the  ratios  in  1  and  2  above  were  based  on  current  ATS 
acquisition  practices;  and  therefore,  the  results  represent  a  No  Change,  Option-l  strategy 
cost  projection. 

The  projected  ATS  acquisition  savings  attributable  to  a  new  investment  strategy 
vfctc  then  computed  by  applying  the  acquisition  strategy  option  factors  to  the  projected 
expenditures. 

The  following  sections  describe  the  DoD  ATS  investment  strategy  evaluated, 
characterize  the  investment  strategy  model  used  in  this  analysis,  and  summarize  tiie  results. 

6.1  INTEGRATED  ATS  INVESTMENT  STRATEGY 

The  integrated  ATS  investment  strategy  applied  in  this  analysis  adopts  an  approach 
which  capitalizes  upon  existing  and  projected  standards:  ATS  Families,  ATS 
Specifications,  and  a  Software  Environment  The  most  significant  influence  is  based  on  an 
approach  that  applies  a  DoD-wide  policy  to  use  designated  standard  ATS  families  to  meet 
testing  needs.  The  ATS  families  approach  is  coupled  with  research  and  development 
programs  to  converge  designated  ATS  families  towards  common  hardware  and  software 
standards  at  critical  interfaces. 
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This  investment  strategy  integrates  elements  of  the  individual  option  cost  factors 
that  were  developed  earUer  in  Section  S.1.1.2.,  Cost  Factors  for  ATS  Investment  Strategy 
Options.  Reviews  of  the  case  study  results  showed  the  relative  influence  of  each  option  cost 
factor  to  be  nearly  independent;  therefore,  the  benefits  could  be  combined.  For  the  purposes 
of  this  assessment,  an  ATS  investment  strategy  that  possessed  the  characteristics  illustrated 
in  Figure  20  was  applied. 


Years  (FY) 

Figure  20.  ATS  Acquisition  Investment  Strategy  Cost  Factors 

The  modeling  technique  assumed  application  of  a  new  ATS  investment  strategy 
that  selects  from  designated  DoD  standard  ATS  families.  Efrectivity  of  benefits  would  be 
gradual  over  a  6  year  period.  That  is,  at  the  end  of  this  period  at  least  70%  of  aU  ATS 
investments  would  be  for  DoD  standard  ATS  families,  and  only  30%  of  the  development 
and  procurement  resources  would  go  for  unique  ATS.  This  was  intended  to  reverse  the 
current  trend  found  during  this  study  where  most  ATS  R&D  and  procurement  funds  go 
directly  to  weapon  program  managers  (PMs),  who  pursue  unique  ATS  acquisitions.  Over 
70%  of  current  development  and  procurement  resources  are  for  unique  ATS  and  about  25% 
are  for  standard  families  (CASS  and  IFTE).  Less  than  5%  are  devoted  to  standard  ATS 
family  modernization. 

The  applicable  percentages  were  iq)plied  to  the  identified  ATS  investment  strategy 
cost  factors  on  a  linear  scale  beginning  in  FY93  at  30%  and  increasing  to  the  final  70%  in 
FY99.  Therefore,  the  curves  represented  by  Figure  20  illustrate  the  assumed 
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implementation  for  the  benefits  postulated  for  Options-Il,  ID,  and  IV  (Common  ATS 
Family,  Common  ATS  Specifications,  and  Common  SW  Environment  re^ctively).  As 
indicated,  the  assumptions  used  to  develop  these  strategy  cost  factors  were  based  on  earlier 
study  results  and  their  magnitude  gradually  ramped  up  to  a  nominal  level  of  weapon  system 
applicability  over  several  years.  The  following  provides  more  details  of  the  assumptions 
used  to  develop  the  ratios  (projected  savings  divided  by  the  expense  category  costs  of  a 
peculiar  ATS)  illustrated  in  Figure  20. 

ATE  Development  Assumes  95%  of  common  ATS  family  development  is  shared 
across  applications.  The  application  curve  is  based  on  an  assumption  that  up  to  70%of  ATS 
acquisitions  will  be  influenced  by  the  strategy  change.  The  implementation  efiectivity  is 
phased  in  linearly  over  a  four-year  period. 

ATE  Production.  Assumes  both  a  production  learning  curve  and  a  quantity 
reduction  benefit  The  application  curve  couples  a  15%  savings  with  a  50%  quantity 
i  tquirement  reduction.  Both  of  these  savings  benefits  ramp  up  linearly  to  70%  of  the 
systems  over  the  same  four-year  period. 

TPS  Development.  Assumes  near-term  development  costs  will  be  reduced  by  10% 
due  to  common  operating  systems  and  station  interfaces,  and  ramps  up  to  70%  over  4  years. 
Also  assumes  a  longer  term  50%  cost  reduction  attributed  to  improved  TPS  development 
and  support  environments.  The  implementation  of  this  second  benefit  is  delayed  4  years 
and  ramps  up  to  70%  over  5  years. 

TPS  Production.  Assumptions  are  identical  to  those  used  in  the  ATE  Production 
expense  category,  except  that  the  production  learning  curve  savings  portion  is  applicable 
to  only  50%  of  the  systems.  This  difference  reflects  the  fact  that  there  is  a  greater  potential 
for  unique  ITAs/IDs. 

6.2  INVESTMENT  ANALYSIS  MODEL  ELEMENTS 

The  main  framework  of  the  ATS  investment  analysis  model  is  based  on  automatic 
test  system  to  weapon  system  cost  ratios,  which  when  applied  to  total  DoD  budgets,  will 
produce  projections  of  annual  ATS  expenditures.  The  total  DoD  projected  annual  ATS 
costs  are  rq>resented  by  the  simple  summation  of  estimated  Service  annual  peculiar  ATS 
costs  combined  with  the  planned  common  ATS  development  and  acquisition.  Potential 
savings  are  computed  by  applying  the  individual  ATS  acquisition  investment  strategy  cost 
factors,  discussed  in  the  previous  section,  to  the  estimated  Service  annual  peculiar  ATS 
costs  and  comparing  the  results. 


Because  weapon  system  commodity  areas  (aircraft,  missiles,  etc.)  exhibit  different 
ATS  ratio  characteristics,  this  analysis  was  divided  up  into  multiple  weapon  system 
groupings.  The  following  weapon  system  commodity  areas  were  selected:  missiles  and 
weapons;  aircraft  Army,  Navy,  and  Air  Force;  weapons  canier/tactical  vehicle;  ships; 
Marine  weapon  systems;  and  other.  The  ratios  of  ATS  production  to  weapon  system  costs 
discussed  in  Section  4.3.2  (and  presented  in  Table  22)  were  used  to  compute  estimated  ATS 
total  expenditures. 

The  study  had  not  focused  nor  collected  detailed  data  on  several  other  weapon 
system  commodity  areas  needed  to  form  a  complete  model.  Therefore,  the  ratio  for  Marine 
weapon  systems  was  approximated  by  the  ratio  computed  for  Navy  aircraft,  and  all  other 
weapon  system  ratios  were  assigned  the  lowest  value  observed.  This  was  considered 
reasonable  because  the  2.66%  value  fell  close  to  the  median  of  computed  ratios,  and  Marine 
weapon  systems  would  include  a  mix  of  commodity  types  that  crossed  the  spectrum.  The 
lowest  ratio  observed  was  0.5%  and  it  was  assigned  specifically  to  ships  (represented  as 
Ship  Building  and  Conversions  in  Navy  budgets)  and  generally  to  all  other  weapon  system 
commodity  areas.  The  ATS  apportionment  ratios  described  in  Section  4.3.2  (and  presented 
in  Table  21)  were  used  to  estimate  costs  by  individual  ATS  expense  categories.  Any 
differences  in  the  computed  apportionment  ratios  and  actual  ratios  for  individual  weapon 
systems  would  only  influence  the  cost  distribution  across  the  expense  categories,  but  would 
not  alter  the  computed  total  costs. 

The  investment  strategy  model  was  implemented  in  a  spreadsheet  format  and  its 
general  characteristics  are  illustrated  in  Figure  21.  Summaries  of  DoD  weapon  system 


Weapon 
System 
(By  Type) 

Summary  of 
EstDoDWS 
Procuremeat 
$’s 

Ratios 

Projected  ATS 
Expenditures 
(Without 
Strategy 
Change) 

ATS 

Cost 

Adjust* 

ments 

Strategy 

Cost 

Factm- 

Ratios 

Projected  ATS 
Savinp 
(With  Invest¬ 
ment  Strategy 
Change) 

TPS 

ATE 

Missile 

WCTV 

• 

• 

• 

Fighter 

e 

.2 

i 

Development 

Production 

Development 

•••  (Devel)»»» 

•••  (Prod)  ••• 

Total  Savings 

Figure  21.  Pictorial  Summary  of  the  Spread  Sheet  Elements  Used  To  Model  DoD 
ATS  Expenditures  and  Potential  Savings 
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(represented  by  WS  in  this  figure)  procurement  costs  were  divided  by  commodity  area  and 
budget  year  across  a  period  from  FY80  to  FY99.  The  data  is  a  combination  of  historical 
DoD  appropriations  as  well  as  current  and  projected  budget  plans.  The  values  used  are 
presented  in  Table  44. 

The  projected  ATS  expenditures  were  corrected  and  adjusted  by  subtracting  CASS 
and  IFTE  related  ATE  development  and  production  costs  (both  actuals  and  projected)  and 
the  CASS  estimated  TPS  off-load  costs.  Table  45  summarizes  the  ATS  cost  adjustments. 
The  new  ATS  acquisition  strategy,  with  its  ratios  presented  in  Figure  20,  was  applied 
against  all  remaining  projected  ATS  expenditures.  The  total  savings  represent  the  annual 
sum  of  potential  cost  avoidance  ATS  savings  across  all  commodity  areas.  The  following 
section  summarizes  the  results  of  the  ATS  investment  analysis. 

63  ATS  INVESTMENT  ANALYSIS  RESULTS 

This  section  presents  the  results  of  the  ATS  investment  analysis  and  reveals  that 
DoD  could  save  over  $1.5  billion  out  of  an  additional  $11  billion  ATS  investment  from 
FY93  to  FY99.  This  section  begins  by  characterizing  the  total  ATS  investnrent  and 
comparing  the  results  with  other  independent  analyses,  and  concludes  with  a  summary  of 
potential  aimual  savings. 

6  J.1  Assessment  of  DoD  ATS  Investment 

The  projected  total  DoD  aiuiual  ATS  expenditures  computed  by  the  investment 
model  are  presented  in  Table  46.  The  data  results  for  FY80  through  FY91  are  based  on 
historical  budget  information,  results  for  FY92  are  based  on  transitional  (actual  and 
estimated)  budget  information,  and  results  for  FY93  through  FY99  are  based  on  budget 
projections.  This  data  along  with  4  data  points  from  Frost  and  Sullivan,  Inc.  reports  (three 
from  [FROST  1983]  and  one  from  [FROST  1990])  are  displayed  in  Figure  22. 

The  Frost  and  Sullivan  totals  include  spare  parts  and  some  operating  and  sustaining 
engineering  costs  not  included  in  the  ATS  investment  model  estimates.  Upon  review  of  the 
Frost  and  Sullivan  data,  it  appears  that  their  numbers  reflect  all  potential  contract  line  items 
that  include  ATS,  even  if  the  contract  were  terminated  or  a  line  item  was  not  exercised. 
Finally,  the  dollar  amounts  indicated  in  the  Frost  and  Sullivan  data  were  associated  with 
the  year  of  award  (or  execution),  whereas  the  ATS  investment  model  reflects  the  estimated 
costs  by  budgeted  year. 
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Table  44.  Weapon  System  Procurement  Costs  by  Commodity  Areas  (FY93$>Millions) 


Table  45.  ATS  Standard  Famiiy  Cost  Adjustment  (FY93$*Millions) 


Expense  Area 

FY93 

FY94 

FY95 

FY96 

FY97 

FY98 

FY99 

IFTER&D 

2.1 

3.6 

2.8 

mg 

2.6 

2.5 

2.5 

CASS  R&D 

17.4 

18.5 

192 

19.9 

20.5 

212 

22.0 

IFTE  PROD  (BSTF) 

21.6 

41.4 

35.6 

31.9 

232 

20.4 

16.9 

CASS  PROD 

196.3 

169.0 

165.6 

164.7 

161.9 

159.8 

157.7 

CASS  TPS  R&D  and 
PROD  CTPS  Off-load 
Estimated  Cost) 

142.0 

142.0 

142.0 

142.0 

142.0 

142.0 

142.0 

Table  46.  Estimates  of  DoD  ATS  Expenditures  From  FY80  To  FY99 


FY 

m 

81 

82 

83 

84 

85 

86 

87 

88 

89 

Constant 

FY93$inM 

1,652 

2,030 

2,627 

3,006 

3,329 

3,548 

3,170 

2,674 

2,317 

2,447 

FY 

lEl 

91 

92 

93 

94 

95 

96 

97 

98 

99 

Constant 

FY9^inM 

■ 

1,776 

1,448 

1,451 

1,429 

1292 

1,350 

1,331 

1,310 
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Maricet  Survey  of  DoD  ATS  Expenditures 

If  q>ares  and  other  support  dollars  were  added  to  the  model  estimates  during  the 
build-up  years  of  1981  through  1983,  the  model  would  have  very  close  correlation  with  the 
Frost  and  Sullivan  data.  The  1989  numbers  from  Frost  and  Sullivan  appear  to  be  high,  and 
this  can  be  attributed  to  several  factors:  spares  costs  not  addressed  by  the  model,  and 
contract  line  items  not  exercised  during  the  build-down  period. 

The  authors  also  concluded  that  the  model  underestimates  some  of  the  weapon 
system  commodity  areas,  and  looked  to  the  Frost  and  Sullivan  data  to  identify  the  most 
likely  areas.  Since  the  ATS  investment  model  identifies  ATS  development  separate  from 
production  (because  DoD  budgets  for  R&D  and  production  are  separate),  the  assumption 
was  made  diat  the  R&D  estimates  would  be  distributed  in  the  same  proportions  as  the 
production  estimates.  Based  on  this  assumption,  the  relative  Frost  and  Sullivan  ATS 
expenditures  were  compared  with  the  ATS  investment  model  percentages  of  expenditures 
for  1989  by  commodity  type.  The  results  of  this  comparison  are  presented  in  Table  47.  We 
noted  that  the  Marine  commodity  (3%)  category  was  not  separately  identified  and 
suspected  that  it  is  evenly  distributed  across  the  other  commodity  areas.  The  authors  also 
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noted  that  the  ATS  expenditure  percentages  for  both  ships  and  the  other  commodity  areas 
were  at  least  twice  as  much  as  those  estimated.  A  potential  single  source  of  error  that  might 
have  produced  this  difference  between  the  Frost  and  Sullivan  data  and  the  model  was  the 
very  conservative  ATS  production  to  weapon  system  cost  ratio  (0.5%)  that  was  assumed 
due  to  a  lack  of  definitive  information.  This  would  imply  that  the  ATS  study  investment 
strategy  model  underestimated  these  two  commodity  areas  for  FY89  by  as  much  as  $300 
million  (constant  FY93$). 

Table  47.  Comparison  of  Commodity  Cost  Distribution  Between  the  Model  and  an 

Industry  Market  Survey 


Commodity  Area 

ATS  Investment 
Model  -  Cost 
Distribution 

Frost  &  Sullivan 
Data  -  Cost 
Distribution 

Aircraft 

65% 

59% 

Missiles 

20% 

16% 

WTCV 

3% 

4% 

Ships 

3% 

9% 

Marine 

3% 

(not  separated) 

Other 

6% 

12% 

Given  the  differences  noted  above,  the  ATS  investment  model  conservatively 
^  estimates  the  total  DoD  ATS  development  and  production  expenditures.  The  greatest 

source  of  ATS  investment  model  error  is  believed  to  originate  from  the  very  conservative 
ratio  assumption  for  ships  and  other  commodity  areas. 

^  63.2  Assessment  of  Potential  ATS  Savings 

Singly  applying  the  ATS  investment  model  using  the  cost  factors  for  the  proposed 
ATS  investment  strategy  and  adjusting  to  then-year  (TY)  dollars  resulted  in  an  estimated 
savings  of  approximately  $1.9  billion  (TY  Dollars)  from  FY93  to  FY99.  However,  this 
0  approach  ignores  additional  R&D  and  sustaining  engineering  that  might  support  die  ATS 

family  strategy  approach.  Therefore,  a  more  conservative  approach  was  considered,  and  the 
results  are  presented  in  Table  48.  In  this  approach  additional  R&D  and  sustaining 
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engineering  funds  are  dedic^ed  to  efforts  that  support  convergence  of  ATS  families 
towards  common  hardware  and  software  standards  at  critical  interfaces. 


Table  48.  Summary  of  Potential  Saving  If  a  New  ATS  Investment  Strategy  Is 
Adopted  (Then* Year  Dollars  in  Millions) 


FY93 

FY94 

FY95 

FY96 

FY97 

FY98 

FY99 

SUM 

Pre-Strategy  Investment 

1.448 

1.497 

1523 

1.531 

1.532 

1J59 

1.584 

10674 

Pre-Plus  Up  Savings 

(28) 

(57) 

(173) 

(284) 

(370) 

(473) 

(494) 

(1,879) 

Common  R&D  Plus  Ups 

21 

32 

44 

54 

56 

64 

78 

349 

IFTE  Plus  Up 

5 

5 

5 

6 

6 

6 

6 

39 

CASS  Plus  Up 

10 

10 

11 

11 

11 

12 

12 

77 

Man-Mobile 

2 

10 

21 

22 

23 

23 

24 

126 

A  Better  Test  Env. 

2 

4 

4 

11 

11 

12 

12 

57 

Next  Generation 

2 

2 

2 

4 

5 

12 

24 

51 

NET  SAVINGS 

(7) 

(25) 

(129) 

(230) 

(315) 

(409) 

(415) 

(1430) 

New  R&D  requirements  were  identiried  for  five  areas:  IFl  t  advances,  CASS 
advances,  man-mobile  ATS  technology,  a  better  test  environment,  and  next  generation 
ATS  requirements.  These  new  R&D  requirements  were  shown  as  common  R&D  plus  ups 
(or  budget  additions)  in  this  table,  and  were  intended  to  continue  the  consnvative  nature  of 
this  analysis.  These  R&D  plus  ups  in  five  areas  were  intended  to  stimulate  convergence  of 
ATS  families  towards  common  hardware  and  software  standards  at  critical  interfaces. 
Some  of  the  plus  up  values  in  this  table  do  not  add  up  to  the  displayed  totals  due  to  rounding 
error.  The  IFTE  and  CASS  were  both  increased  by  $S  and  $10  million  annually  (adjusted 
to  estimated  TY$  in  the  table).  These  additions  were  to  address  new  common  testing 
requirements  and  other  potential  P^I  needs  of  the  future.  The  man-mobile  plus  up  was 
intended  to  address  portable  and  downsizing  requirements  of  the  future;  needs  of  this  nature 
have  been  identified  by  both  the  Air  Force  and  the  Marines.  A  better  test  environment  plus 
up  is  needed  to  address  the  capabilities  and  benerits  that  were  identified  earlier  under 
Option  rv.  Common  SW  Environment  Development  funding  will  be  needed  to  achieve 
these  capabilities.  Finally,  the  next  generation  plus  up  supports  research  and  development 
efforts  needed  to  plan  for  the  next  generations  of  common  ATS  families. 

A  graphical  sununary  of  the  results  is  presented  in  Figure  23.  Here  again  the  results 
are  shown  as  constant  FY93$  to  be  consistent  with  the  earlier  figure.  The  dark  bars 


120 


illustrate  the  DoD  annual  ATS  investment  baseline  computed  using  the  developed  model. 
The  baseline  in  this  figure  (same  curve  as  shown  in  previous  figure)  is  defined  to  represent 
the  aimual  investment  without  any  ATS  investment  strategy  change.  The  white  bars 
illustrate  the  new  aiuiual  ATS  investment  based  on  the  estimated  net  savings  from  Table 
48,  only  here  they  have  been  converted  back  to  constant  FY93$. 


Figure  23.  Comparison  of  Total  DoD  A're  Acquisition  Cost  Estimates  Under 
Current  (Ba^ine)  Practices  and  a  Proposed  New  Investment  Strategy 
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7.  POLICY,  PROCESS,  FUNDING,  AND  ORGANIZATIONS 


Dispersed  ATS  investment  decisions,  both  within  as  well  as  across  Service  and 
weiq)on  program  lines,  are  viewed  as  a  major  contributing  factor  to  DoD  ATS  proliferation. 
Historically,  most  DoD  ATS  investments  have  been  made  in  a  decentralized  fashion  by 
individual  program  and  depot  managers.  This  has  resulted  in  acquisition  of  many  different 
ATS  with  similar  capabilities.  Since  ATS  is  a  key  element  of  maintenance,  defense 
contractors  have  historically  gained  a  sole  source  support  foothold  by  control  of  program 
peculiar  ATS.  Weapon  system  program  logistics  schedules  are  greatly  influenced  by  ATS 
availability.  Many  programs,  such  as  the  F-16  and  Apache,  have  developed  and  upgraded 
weapon  unique  ATS  over  a  long  life  cycle;  the  reprocurement  and  support  for  this  ATS  has 
been  essentially  captured  by  one  contractor  source. 

When  the  ATS  decisions  have  been  decentralized  to  the  weapon  manager,  a 
weapon-unique  approach  was  most  often  selected.  The  study  data  and  analyses  showed 
that,  in  areas  where  there  has  been  clear  central  authority  for  the  ATS  acquisition  and 
funding  decisions,  unique  duplicative  ATS  have  been  minimized  and  standard  ATS  have 
been  successfully  used  to  achieve  cost  and  logistics  benefits. 

The  following  sections  summarize  die  principal  ATS  policies,  and  describe  the 
predominant  ATS  management  process  and  funding  flow  for  DoD  organizations  found  to 
be  involved  with  ATS  acquisitions.  The  analysis  begins  with  discussions  of  current  DoD 
policy  and  extends  to  implementation  within  the  Services. 

•  DoD  has  no  specific  ATS  policy  and  the  Services’  ATS  policies  and  programs 
have  been  largely  independent 

•  Opportunities  to  leverage  common  investments  both  across  and  within  the  Ser¬ 
vices  have  been  lost  and  duplicative  ATS  investments  have  been  common. 

•  Consequently  DoD  is  seen  as  having  no  guidance,  process  mechanism,  or  orga¬ 
nizational  approach  to  ensure  that  ATS  are  managed  and  acquired  efficiently 
across  the  Services. 

•  The  Army  and  Navy  Service  level  policies  and  processes  have  evolved  towards 
limited  centralized  ATS  management  The  Air  Force’s  attempt  to  institutionalize 
a  standard  ATS  management  process  (MATE)  was  unsuccessffil.  The  Air  Force  is 
now  in  the  process  of  developing  a  new  policy  and  organizational  structure. 


7.1  DOD 

There  is  no  DoD-level  policy  or  process  limiting  ATS  acquisition  decisions  or 
fostering  cross-Service  ATS  use.  ATS  are  not  specifically  identified  in  the  Department’s  ® 

directives  or  instructions  and  fall  under  the  general  guidelines  for  all  systems  acquisitions. 

DoDD  5000.1,  Defense  Acquisition,  February  23, 1991.  Establishes  a  disciplined 
management  approach  for  all  systems  acquisitions,  including  ATS.  ^ 

DoDI  5000J1,  Defense  Acquisition  Management  Policies  and  Procedures, 

February  23,  1991.  Part  6,  Section  D,  Computer  Resources,  authorizes  the  ATLAS 
programming  language  for  ATE;  and  Part  7,  Section  A,  Integrated  Logistics  Support, 
requires  that  support  structure  elements  be  appropriately  considered  and  acquired  to  ensure  % 

a  supportable  system. 


Memorandum,  Under  Secretary  of  Defense  (Research  and  Development), 
ATE  Language  Standardization,  1979.  The  DoD  ATE  Language  Standardization 
Committee  (DALSCOM),  established  by  this  memorandum,  is  chartered  to  mature, 
implement  and  refine  the  Institute  of  Electrical  and  Electronic  Engineers  (IEEE)  ATLAS 
for  DoD  use.  This  is  the  only  other  DoD  structured  involvement  in  ATS  policy  and  it 
addresses  ATE  language  standardization. 

Joint  Services  Automatic  Testing  Panel  (JSATP),  and  its  predecessor,  the  Joint 
Commanders  Panel  on  Automatic  Testing  (originally  established  in  1987).  Although 
not  under  the  auspices  of  the  DoD,  Service  members  have  made  an  attempt  to  coordinate 
development,  selection,  acquisition  and  logistics  policies  via  these  two  groups.  The  JSATP, 
through  its  sub-panels  and  Executive  Board,  has  had  some  success  by  improving 
communication  and  information  exchange,  but  its  lack  of  authority  has  limited  its  ability  to 
significantly  address  proliferation  problems  identified  in  this  study  and  earlier  audit 
reports. 


7.2  AIR  FORCE 

Historically,  the  Air  Force  has  not  had  a  Service-level  ATS  standardization  policy 
and  has  essentially  pursued  unique  ATS  solutions  for  each  weapon  system.  To  cite  some 
current  examples,  the  Air  Force  is  pursuing  new  ATS  acquisitions  for  the  F-22,  the  F- 15 
(downsized  tester),  the  F- 16  intermediate  (downsized  tester),  and  the  F- 16  depot;  none  are 
requiring  use  of  the  same  ATS  or  a  subset  of  the  same  ATS,  although  several  areas  of 
requirements  overlap.  The  AF  Systems  Command/Logistics  Command  (AFSC/AFLC) 
directed  approach,  called  MATE,  imposed  operating  system  software  and  a  set  of  standards 
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to  guide  weapon  program  ATS  acquisition.  The  MATE  policy  was  not  effectively 
enforced.  By  the  Air  Force’s  accounting,  MATE  was  applied  in  roughly  15%  of  the  Air 
Force  weapon  programs,  primarily  in  depot  operations.  The  following  summarize  the 
principal  Air  Force  ATS  policies  in  place  at  the  time  of  this  study. 

Air  Force  Relation  800-12,  Acquisition  of  Support  Equipment,  August  18, 
1989.  This  regulation  excludes  ATS.  Air  Force  activities  that  acquire,  modify,  replace,  or 
support  systems  requiring  ATS  were  to  follow  their  respective  regulations  for  meeting  the 
requirements. 

Memorandum,  Deputy  Assistant  Secretary  of  the  Air  Force  (SAF/AQK), 
Policy  for  Use  of  ADA  and  ATLAS  for  ATS,  February  18, 1992.  This  memorandum 
states  diat: 

a.  New  programs  are  encouraged  to  develop  TPSs  consistent  with  Ada  Based 
Environment  for  Test  (originally  known  as  ABET,  but  due  to  a  conflict  of  names 
is  now  known  as  ABBET  or  A  Broad  Based  Environment  for  Test)  unless  the 
cost  or  risk  is  significant 

b.  Programs  intending  to  use  ABET  must  submit  an  implementation  plan  for 
approval;  this  is  intended  so  that  the  Air  Force  is  able  to  keep  ABET 
implementations  consistent 

c.  ATE  that  is  not  compliant  with  ABET  is  to  be  MATE  compliant  and  where  ATE 
is  neidier,  a  MATE  waiver  is  required  per  Air  Force  Systems  Conunand/ 
Logistics  Command  (AFSOAFLC)  Regulation  800-23  [MOSEMANN  1992]. 

AFSC/AFLC  Regulation  800-23,  Policy  for  MATE,  January  25,  1984.  This 
regulation  states  that  ATS  for  logistic  suppoit  will  use  MATE  system  procedures  unless  a 
waiver  is  granted.  Waivers  are  to  be  considered  in  terms  of  the  increased  life  cycle  costs  of 
the  prime  weapon  system,  life  cycle  cost  increases  of  the  ATS  with  no  offsetting  cost 
benefits,  incompatibility  of  MATE  system  technology  with  the  prime  weapon,  and 
compatibility  with  the  operational  need  date.  The  acquiring  major  command  has  the  final 
waiver  authority  under  this  regulation. 

Air  Force  ATS  R&D  and  procurement  funding  and  the  acquisition  responsibility 
have  been  widely  dispersed  in  program  offices  and  at  Air  Logistics  Centers.  As  this  study 
was  concluding  the  Air  Force  published  a  new  Service-level  policy  letter  (July  30,  1992) 
stating  the  following;  “It  is  the  Air  Force  policy  to  use  DoD  inventoried  ATS  or  common 
commercial  ATS.  Essentially,  peculiar  ATS  should  be  developed  and  acquired  on  an 
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exception  basis.  Funding  and  ultimate  responsibility  for  ATS  applicable  to  a  weapon 
system  must  rest  with  the  weapon  system  program  director.”  The  letter  requires  that  a 
central  ATS  office  be  established  to  implement  the  policy;  this  office  will  cochair  an  ATS 
acquisition  panel  with  the  weapon  manager  (“common”  and  “peculiar”  as  used  in  the  policy 
were  undefined.)  Under  the  new  organization,  the  new  ATS  System  Program  Director 
(SPD)  cochairs,  with  the  applicable  Weapon  SPD,  an  ATS  Acquisition  Panel,  The  Program 
Executive  Officer  (PEO),  Designated  Acquisition  Commander  (DAC),  or  the  Air  Force 
Acquisition  Executive  (AFAE),  as  appropriate,  resolve  conflicts.  Planning  for  the  Air 
Force’s  centralized  ATS  organization,  along  with  all  deEnitions  of  its  functions  and 
authority,  was  still  underway  as  this  report  was  written. 

A  chart  of  the  current  ATS  acquisition  process.  Figure  24,  was  prepared  by  the  Air 
Force.  The  syntax  used  in  this  figure  is  as  follows: 

a.  Requirements  enter  a  process  block  from  the  left 

b.  Constraints  on  the  process  enter  the  block  from  the  top  (e.g.,  funding  and 
management). 

c.  Performing  and  participating  activities  that  perform  work  enter  the  block  from 
the  bottom. 

d.  Products  of  the  process  leave  the  block  from  the  right. 

Simultaneous  with  the  development  of  this  new  ATS  policy,  the  Air  Force  is 
undergoing  a  substantial  reorganization  of  the  acquisition,  support,  and  operational 
commands.  Sweeping  changes  in  operations  could  influence  Air  Force  ATS  requirements 
and  implementation  policy.  For  example  die  Air  Force  is  investigating  two-level  field 
maintenance  changes,  is  implementing  composite  wing  concepts,  and  is  discussing 
regional  I-level  maintenance  concepts. 

7.3  NAVY 

The  Navy  has  taken  steps  to  acquire  and  manage  ATS  as  a  cross-weapon 
commodity.  Naval  aviation  implemented  earlier  generation  ATS  families  for  a  subset  of  its 
operations  in  the  1970s,  most  notably  using  the  VAST  ATS  in  carriers  and  depots.  During 
the  mid-1980s,  the  Navy  aviation  community  developed  their  modem,  comprehensive 
Service  ATS  family,  CASS.  The  Navy  aviation  resolve  to  develop  and  deploy  the  CASS 
ATS  family  is  partially  attributable  to  the  Service-wide  ATS  policy  and  a  centralized  ATS 
development  and  procurement  orgaruzation.  Tlie  Navy  has  a  centralized  organization, 
responsible  for  ATS  acquisition  policy  and  practices;  however,  it  has  not  been  fully 
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Figure  24.  Air  Force  ATS  Acquisition  Process 

implemented  by  all  of  the  Navy’s  major  Systems  Commands.  As  of  diis  study  a  Navy-wide 
implementation  process  was  being  defined.  The  foUowing  summarize  the  principal  Navy 
ATS  policies  in  place  at  the  time  of  this  study. 

Secretary  of  the  Navy  Instruction  3960.6,  Department  of  the  Navy  Polity  and 
Responsibility  for  Test,  Measurement,  Monitoring,  Diagnostic  Equipmoit  and 
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Systems,  and  Metrolc^  and  Calibration,  October  12, 1990.  Under  this  nuindate,  ATE 
is  to  be  standardized  as  much  as  possible  and  CASS  is  designated  as  the  Navy’s  standard 
ATE.  Under  this  policy,  development  and  acquisition  of  non-CASS  ATE  requires  Assistant 
Secretary  of  the  Navy  for  Research,  Development  and  Acquisition  (RD&A)  approval. 

Naval  Air  Systems  Command  (NAVAIR)  Instruction  13630.2A,  Introducing 
the  CASS  to  Naval  Aviation  Maintenance,  March  22, 1991.  This  instruction  establishes 
the  command’s  policy  for  optimizing  the  use  of  CASS  and  associated  TPSs  for  all 
NAVAIR  programs  and  states  that  all  electronic  weapon  systems/subsystems  are  to  be 
designed  in  consideration  of  the  following: 

a.  Ease  of  testing  and  compatibility  with  CASS. 

b.  CASS  or  compatible  equipment  will  be  specified  as  the  factory  test  equipment  to 
be  used  at  a  development  or  manufacturing  facility  for  acceptance  testing. 

c.  The  CASS  is  to  be  the  target  system  for  all  intermediate  and  depot  level  ATE 
requirements. 

d.  Waiver  approval  is  required  if  instances  arise  where  CASS  is  determined  not  to 
be  the  optimum  ATE  support  solution  based  on  1)  life  cycle  cost  comparison  to 
alternatives,  2)  program  deployment/activation  schedules,  3)  Technologies  not 
within  CASS  capability,  or  4)  facility/mobilization  impacts. 

Naval  Sea  Systems  Command  (NAVSEASYSCOM)  Memorandum,  Test, 
Measurement,  Monitoring,  Diagnostics  Equipment  and  Systems  and  Metrology  and 
Calibration,  January  28, 1991.  This  memorandum  directs  the  following: 

a.  Use  of  CASS  when  feasible  for  production,  depot,  and  Held  testing. 

b.  Use  of  MIL-STD-2165,  Testability  Program  for  Electronic  Systems  and 
Equipment,  to  ensure  effective  testability  is  incorporated  into  the  design  of  new 
systems. 

c.  The  design  of  weapon  platforms  and  systems  diagnostics  as  part  of  system 
engineering. 

d.  Integration  of  system  design  with  the  Logistics  Support  Analysis  (LSA). 

Weapons  &  Combat  Systems  Directorate  Guidance  &  Policy  Paper  No.  87-13 

dated  May  26,  1987.  An  earlier  policy  was  published  by  the  Deputy  Commander  for 
Weapons  and  Combat  Systems  to  ensure  implementation  of  CASS  and  the  reduction  of 
specialized  ATE.  This  policy  was  intended  to  make  CASS  the  preferred  alternative  when 
ATE  is  to  be  utilized. 


# 


# 
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The  Naval  Space  and  Warfare  Systems  Command  (SPAWAR)  Maintenance 
Policy  Instruction  4600.17,  (Not  Sipied).  This  instruction  was  awaiting  final  signature  as 
this  study  was  concluding.  It  mirrors  the  Secretary  of  the  Navy  Instruction  3960.6.  llie 
interim  policy  contained  in  the  Consolidated  Automated  Support  System  (CASS)  Usage 
memorandum,  dated  January  18, 1991,  adheres  to  the  Secretary’s  instruction. 

The  Navy  ATS  program  is  organized  around  PMA-260  located  at  Naval  Air 
Systems  Command,  Arlington,  VA.  PMA-260  has  full  CASS  ATS  development  and 
procurement  responsibility.  He  is  supported  by  AIR-S52,  Support  Equipment  Division,  via 
a  matrix  organization  authority  structure  for  requirements  determination  and  the  CASS 
Class  Desk  technical  systems  engineering  support  Assistant  Program  Manager  for 
Logistics  (APML)  engineering  support  for  the  system  is  provided  through  the  NAVAIR- 
04  organization.  The  Program  Manager  reports  to  the  Secretary  of  the  Navy  via  the 
Commander,  Naval  Air  Systems  Command,  through  a  flag  level  NAVAIR-01  PEO 
structure.  The  CASS  charter  and  matrix  stracture  currently  implement  Navy-wide  CASS 
R&D  and  procurement  NAVAIR-wide  Common  Support  Equipment  (CSE)  R&D  and 
procurement  and  NAVAIR-wide  weapon  peculiar  ATS  and  TPS  management 

Currently,  NAVAIR  is  in  the  process  of  implementing  a  command- wide 
downsizingA'eorganization  plan  that  combines  the  CASS  Program  Manager  PMA-260  and 
the  AIR-SS2  Support  Equipment  Division  into  a  single  organization.  This  organization  will 
maintain  the  CASS  (including  re-host  TPS  programs)  and  CSE  development  and 
acquisition  as  focus  areas,  but  transfers  weapon-peculiar  support  equipment  (PSE)  and  TPS 
management  and  procurement  to  the  weapon  system  program  managers.  The  new  office 
will  retain  the  responsibility  for  integrating  the  total  aviation  support  equipment  program 
policies  and  processes. 

CASS  is  funded  under  Program  Element  Number  PE64215N  Support  Equipment 
( CASS)  for  RDT&E  and  APN-7  for  procurement  of  all  production  items.  These  funds  along 
with  the  off-load  TPS  funds  are  coordinated  and  controlled  by  the  CASS  Program  Manager 
through  his  Chief  of  Naval  Operations  (OPNAV)  Program  Sponsor  .Emerging  system  TPSs 
are  the  funding  responsibility  of  the  individual  weapon  system  program  manager. 
However,  Support  Equipment  Project  Officers  (SEPOs)  within  AIR-552  act  as  TPS 
requirements  and  acquisition  managers.  SEPOs  determine  CASS  requirements  and  report 
directly  to  AIR-5S2  via  a  matrix  organization.  CASS  planning  is  accomplished  through  an 
AIR-552  introduction  planning  document  called  the  CASS  Introduction  Plan  (CIP)  which 
reflects  those  e^orts  required  to  assure  timely  introduction  of  new  weapon  systems  support 
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to  CASS  and  the  coordinated  retirement  of  existing  test  equipment.  The  CIP  details  all 
programmatic  information  and  also  provides  the  number  and  type  of  CASS  stations  and 
ancillary  equipment  required  for  TPS  development  and  operating  fleet  shore  and  afloat  site 
activations.  Transition  efforts  are  being  focused  in  the  following  six  areas: 

a.  Tlie  identihcation  and  support  requirements  of  emerging  avionics  systems. 

b.  The  preparation  of  more  cost  effective  TPS  contracting  methods  and 
procurement  management  procedures  that  are  performance  driven. 

c.  The  identification  of  CASS  station  r«}uirements  to  support  TPS  development. 

d.  The  identification  of  site-specific  CASS  station  configurations  and  fleet  delivery 
schedules. 

e.  The  determination  of  ATE  off-load  priorities,  schedules,  and  resources. 

f.  The  establishment  of  an  interim  support  program  for  specific  UUTs  if  CASS  is 
not  the  immediately  available  solution  for  emerging  high  priority  WRAs/SRAs, 
as  required. 

The  Navy  uses  a  wide  ranging  suite  of  technical  and  management  tools  that  support 
requirements  determination  and  acquisition  processes  for  CASS  and  weapon  system  TPS. 
These  tools  include  the  following: 

a. -  The  System  Synthesis  Model  (SSM)  mapping  tool  is  used  to  define  the  CASS 

configurations  best  suited  for  UUT  support 

b.  The  SSM  workload  tool  is  used  to  better  define  the  optimum  configuration  mix 
and  quantity  of  CASS  stations  at  a  fleet  site. 

c.  The  SSM  also  helps  identify  needed  interface  device  capabilities  and  potential 
CASS  capability  upgrades. 

d.  The  Technical  Risk  Assessment  Guide  is  used  to  determine  TPS  complexity 
factors  that  are  used  as  inputs  to  a  TPS  cost  model. 

e.  The  CASS  Introduction  Plan  data  base  is  used  to  store,  analyze,  and  distribute 
programmatic  and  financial  data  about  all  programs 

A  chart  of  the  current  ATS  acquisition  process.  Figure  25,  was  prepared  by  the  Navy. 


Army’s  resolve  to  develop  and  deploy  the  IFTE  ATS  family  is  partially  attributable  to  its 
Service-wide  organization  for  all  test  and  measurement  equipment  oversight 

Army  Regulation  75(M3,  Army  Test,  Measurement,  and  Diagno^ic 
Equipment  Program,  September  29,  1989.  This  regulation  establishes  Service-level 
policy  and  an  Army-wide  centralized  organization  for  ATE  and  TPS.  It  is  intended  to 
ensure  that  development  and  fielding  of  ATE  is  according  to  the  Army’s  maintenance 
structure.  Under  this  policy,  IFTE  is  designated  as  the  Army  standard  ATE  and  represents 
an  attempt  to  avoid  ATS  proliferation.  The  executive  agent  for  this  Test,  Measurement,  and 
Diagnostic  Equipment  (TMDE)  is  the  Commanding  General,  U.S.  Army  Material 
Command.  The  Deputy  Executive  Director  for  TMDE  serves  as  the  central  focal  point  for 
ATE,  TPS,  and  software  language  requirements  related  to  automated  testing.  In  this 
capacity,  he  develops  and  establishes  ATE,  TPS  and  post-deployment  software  suppon 
policies  and  procedures,  and  reviews  and  approves  waiver  requests. 

Policy  implementation  is  controlled  almost  exclusively  through  a  waiver  process. 
Non-standard  ATE  wUl  not  be  used  in  lieu  of  designated  standard  ATE  without  an 
appropriate  economic  analysis.  System  developers  in  coordination  with  the  Program 
Manager  for  Test,  Measurement,  and  Diagnostic  Equipment  (PM  TMDE),  U.S.  Army 
TMDE  Activity  (USATA),  and  TRADOC  determine  ATE  requirements.  Once  ATE 
requirements  have  been  identifled,  the  system  developer  determines  if  the  designed 
standard  ATE  will  fulfill  requirements,  and  where  they  do  not,  determines  the  feasibility  of 
expanding  capabilities  of  standard  ATE.  If  neither  of  the  above  are  practical,  the  system 
developer  submits  a  waiver  request  documenting  the  case  for  a  non-standard  ATE.  Based 
on  information  provided  in  the  TIMs,  the  most  frequently  used  rationale  has  to  do  with 
weapon  system  schedule  needs  often  associated  with  late  ATS  designation. 

HQ,  Army  Material  Command  Memorandum  AMC-M  750-1,  Automatic  Test 
Equipment  Policy,  August  6, 1991.  This  memorandum  ensures  that  ATE  development 
and  fielding  is  according  to  Army  Regulation  750-43.  This  regulation  designates  the  IFTE, 
a  highly  mobile  or  fixed  site  tester,  to  be  the  Army  standard  ATE  that  will  be  used  for  all 
new  systems  as  well  as  currently  fielded  systems  undergoing  P^I.  The  policy  requires  that 
Army-wide  cost  and  effectiveness  considerations  be  made  as  according  to  Army 
Regulation  750-43,  that  PM  TMDE  be  responsible  for  the  logistic  support  of  ATE  and 
embedded  software,  and  that  the  Major  Subordinate  Material  Development  Commands 
identify  TPS  requirements  not  later  than  milestone  II  of  the  prime  system. 
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Army  PunphlcC  750-43,  Army  Test  Program  Set  Procedures.  This  panfq>hlet 
I^vides  guidance  for  ^tplying  requirements,  acquiution,  development,  and  life  cycle 
nwnagement  of  TPS  in  support  of  Army  Material  Command  systems.  The  Program 
Manager  TMDE  manages  the  implementation  of  general  purpose  TMDE/ATE, 
promulgates  technical  guidance  for  weapon  system  developers,  implements  and  controls 
the  standard  ATE  language,  formalizes  and  publishes  doctrine  for  TPS  development,  and 
develops  policy,  specifications,  and  standards  for  electronic  information  delivery. 

The  Army  has  centrally  organized  its  ATS  program  around  the  TMDE  activity 
located  at  Redstone  Arsenal,  AL.  The  TMDE  organization  is  under  and  reports  to  the 
Conunanding  General  for  the  Army  Material  Command  (CG,AMC).  The  TMDE  charter 
implements  Army-wide  management,  research  and  development,  central  procurements  for 
common  and  tactical  TMDE,  and  the  ATS  waiver  process. 

ATE  requirements  at  all  echelons  of  nuuntenance  are  identified  and  justified  by  a 
complete  system  Repair  Level  Analysis  (RLA).  These  requirements  for  both  hardware  and 
software  are  identified  and  validated  by  die  USATA,  TRADOC,  DESCOM,  and  system 
developer  activities.  After  confirming  ATE  requirements,  a  determination  is  made  if 
requirements  can  be  met  with  the  IFTE  syston.  If  not,  the  feasibility  of  enhancing  DTE 
capability  is  assessed.  If  neither  of  these  alternatives  is  acceptable,  the  system  developer  or 
requiring  activity  will  submit  a  waiver  request  Peculiar  ATE  is  not  substituted  for  IFTE 
without  an  approved  waiver.  Considnadon  of  a  waiver  requires  a  validated  economic 
analysis  ensuring  the  IFTE  system  as  the  baseline  alternative. 

The  requirements  and  schedules  for  the  system  RLA  and  LSA  are  completed  no 
later  than  milestone  n  for  the  prime  weapon  system  under  development  while  individual 
TPS  requirements  are  established  by  the  various  TPS  centers.  The  Program  Manager 
TMDE,  however,  only  provides  policy  guidance  for  die  development,  acquisition,  and 
modemizadon  of  Army  general  purpose  ATE.  Similar  to  the  Navy,  the  USATA  has 
developed  and  maintains  a  Test  Equipment  Management  Infoimadon  System  (TEMIS). 

TPS  acquisidon  is  a  separate  acdon  entirely.  The  priority  assigned  to  the  TPS 
development  and  procurement  is  driven  by  the  end-item  weapon  system  they  support 
Documentadon  for  planning,  development,  acquisidon,  and  maintenance  of  resources  is 
contained  in  the  TPS  Master  Plan  (TPSMP). 

IFTE  R&D  funding  is  budgeted  under  Program  Element  Number  PE  64746A 
Integrated  Family  of  Test  Equipment  (IFTE).  Development  and  procurement  of  all  IFTE 


items  are  budgeted,  planned,  and  coordinated  by  the  DTE  Program  Manager  through  his 
Program  Sponsor.  TPSs  remain  the  funding  responsibility  of  the  individual  weapon  system 
program  manager. 

A  chart  of  the  current  ATS  acquisition  process.  Figure  26,  was  prepared  by  the  Army. 


Figure  26.  Army  ATS  Acquisition  Process 
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8.  ATS  CAPABILITIES  AND  APPLICATIONS 


This  section  describes  the  capabilities  of  proposed  DoD  designated  ATS  standard 
families.  Analyses  of  these  ATS  and  other  in-inventory  ATS  were  performed.  Several  of 
the  weapon  systems  LRUs  were  reviewed  by  the  Services  and  test  requirements  of  the  most 
difficult  to  test  LRUs  were  identified.  The  test  requirements  for  these  LRUs  were  compared 
with  the  test  requirements  capability  envelopes  of  these  ATS. 

Several  current  DoD  ATS  test  cqiabilities  envelopes  were  compared  to  various 
weapon  systems  test  requirements  envelopes  cutting  across  the  Services.  Results  showed 
that  many  existing  DoD  ATS  were  ctqiable  of  meeting  95%  of  the  test  requirements 
envelope  without  test  resource  extension.  These  ATS  had  not  been  specifically  designed  to 
a  DoD-wide  test  performance  requirements  envelope  as  was  the  case  for  CASS  and  IFTE. 
Throughout  the  analysis,  the  data  shows  that  for  the  few  exceptions  where  testing 
limitations  were  identified,  only  minor  additions  to  CASS  and  IFTE  would  be  required  in 
order  to  support  the  analyzed  UUTs. 

8.1  ATS  FAMILY  CAPABILITIES 

CASS  and  IFTE  were  developed  as  general  purpose  ATS  and  designed  to  be 
configured  into  broad  coverage  ATS  families.  The  Army,  Navy,  and  Marines  performed 
extensive  mission  and  test  requirements  studies,  including  assessments  of  the  capabilities 
of  fielded  ATS  and  future  mission  and  test  requirements  across  many  weapon  systems. 
These  studies  resulted  in  a  new  test  capability  requirements  envelope  for  general  purpose 
ATS.  The  ATS  development  programs  for  the  CASS  and  IFTE  met  or  exceeded  the 
advanced  test  requirements  envelope  defined  by  these  studies  [NAEC  1982]  [CECOM 
JULY  83]  [CECOM  DEC  83]  as  well  as  mission  deployment  requirements. 

The  ATS  development  requirements  included  projections  of  future  test 
requirements,  instrument,  and  test  resource  technology  to  assure  that  the  CASS  and  IFTE 
ATS  contractors  deliver  instrument  and  test  resource  technology  at  IOC  (Initial 
Operational  Capability)  to  meet  or  exceed  die  projected  test  envelope  needed. 

Test  resource  vendors  have  delivered  test  resources  to  the  CASS  program  that 
represent  the  state  of  the  art  at  IOC.  Hewlett  Packard  has  indicated  that  all  of  the 
instruments  developed  for  the  CASS  which  were  put  into  the  downsized  MMS  instrument 
packaging  standard  were  designed  and  developed  during  the  last  three  years  to  meet  CASS 
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IOC  ui  1993  [STERN  1992].  The  cuirency  of  this  set  of  general  purpose  test  resources 
cannot  be  exceeded  in  fielded  equipment  today. 

A  number  of  specific  assessments  of  the  DoD  ATS  families  testing  capability 
compared  to  specific  DoD  testing  needs  have  been  reported.  The  following  sections  report 
on  these  assessments.  These  assessments  together  indicate  the  potential  test  capability 
provided  by  the  two  designated  DoD  ATS  standard  families  (CASS  and  IFTE). 

8.1.1  CASS  General  Purpose  Station  Set 

CASS  is  composed  of  four  station  types  and  five  standard  ancillary  units.  The  four 
station  types  are  composed  of  a  core  hybrid  test  capability  (common  low  frequency  analog 
and  digital  capability  available  in  all  station  types)  and  additional  specialized  test  resources 
unique  to  each  of  the  four  stations.  The  four  CASS  station  types  are  the  Hybrid,  CNl 
(Communication,  navigation,  identification),  RF  (radio  frequency,  radar,  EW,  and 
miCTOwave),  and  EO  (electro-optic).  The  five  ancillary  CASS  units  are  (1)  INS  (inertial 
navigation  system),  (2)  ACBI  (advanced  communication  bus  interface),  (3)  PEG 
(pneumatic  function  generator),  (4)  MIL-STD  1397  (Input/output  interfaces  for  standard 
digital  data),  and  (5)  the  printer. 

The  key  attributes  for  each  of  the  principal  ATE  elements  (presented  in  Figure  1 ,  in 
Section  1.3)  are  listed  in  Table  49.  The  CASS  capabilities  for  each  of  these  attributes  are 
summarized  in  Tables  50  to  53. 

The  common  core  test  resources,  column  one  of  Table  54,  are  found  in  all  test 
station  configurations.  The  RF  resources  are  found  in  both  the  RF  and  CNl  configurations 
with  the  exception  of  spread  spectrum  resources,  which  are  only  found  in  the  CNl 
configuration.  Miscellaneous  resources  are  used  in  one  or  more  configurations  and  optional 
ancillary  resources  may  be  used  in  any  configuration.  The  CASS  station  sets  and  optional 
ancillary  units  provide  the  most  complete  general  purpose  automatic  testing  capability  in 
the  DoD  inventory. 

CASS  test  stimulus  and  measurement  capabilities  at  the  front  panel  interface  were 
specified  in  1986  to  take  advantage  of  anticipated  1992  test  instrument  capabilities.  By 
establishing  requirements  at  the  front  panel  interface,  the  CASS  Program  Office 
compensated  for  line-loses,  distortion,  and  noise  that  might  be  introduced  when  integrating 
instruments  and  switches.  Since  all  ATS  architectures  require  interfaces  (with  switches. 


136 


m 


etc.)  b^ween  instruments  and  the  UUT,  the  achieved  CASS  ^)ecifications  rqiresent  the 
leading  edge  of  testing  technology  c^Mbilides  today. 

Table  49.  Key  Attributes  of  ATE  Elements 


iBtafKC 


Geaera]  PuqiKMe 

Electrical 

IniofKeiype 

Progrunmabk 
Power  Supplies 

Mkrowave  Signal 
Interface 

Cooling  Source 
Interface 


Headng  Source 
Interface 


Optical  Bench 
and  Collimator 
Interface 

Motion  Source 
Interface 

Pneumatic 

Interface 

Uut  Structural 
Suppnt  Interface 


Sttamilits 

McasurcBMat 

Digital  Stored 
Pattern 

Digital  Response 
Pattern 

Analog  Waveforms 

Waveform 

Analysis 

Communication 

Waveforms 

Modulators, 

Demodulation 

Microwave 

Signals, 

Modulation, 
Broadband  Noise 

Spectrum 

Analysis,  VUts, 
Power,  Phme 
Demodulation. 

Phase  Noise 

Electro-Optic 

Waveforms, 

Sources 

Optical  Detectm^ 

Powo/Fiequency 

Modulation 

Ihrget  Collimator 

Optical  Detector 
Image  Analysis 

Multiple  Ibrget 
Generators 

Current,  Voltage 
Power,  Noise 

Pneumatic 

Functions 

Pressure/Rate 

1 

Control 

and 

Switch 

Test 

Equipasent 

Opcf^tng 

System 

Power 

Test  Language 

Signal  Matrix 
Switch 

Digital  Test 
Formats 

Stimulus 

Synchronization 

Comjdex 

Waveform 

Measurement 

Synchronization 

Real  Tune 
Software 

Control 

Standard  Buses/ 
ProtoctdsFOr 
Mode  Control 

Simultaneous 

Thread 

Management 

Interface 

Characteristics 

Control 

Mainieiuuice 

Data/Services 

Interface 

UUT 

Synchronization 

Technical 

Manual 

Services 

ID  Mode  Control 
UUT  Mode 

Control 

Self-Test 

Smvices 

Calibration 

Sources 

Self- 

Calibration 

Services 
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Tible  50.  Interface  Characteristics  of  CASS 


L 


iBieiface 

Characteristics 

General  Purpose  Electrical 
Interface  IVpe 

Virginia  Panel,  1486  Total  Quantity.  1500  Volts/Pin. 

76  Power  Paths,  512  Signal  Paths, 

210  Low  Frequency  Pins  @  3  amps  DC  to  10  MHZ 

64 Coax®  500 ma.  DC  to 2  GHZ 

448  Digital  Pins,  DC  to  40  MBIT/Sec 

40  Power  Pins  @  25  amps,  36  Power  Pins  @  10  amps 

Prograiiunable  Power  Supplies 
and  Loads 

Low  Voltage  11  @  900  W/450V/25A 

AC  Power  3  @  55  ->  1200  HZ.  1  ->  135  V.  5A.  675  W. 

1 15/200V  3PH  400  HZ  MIL-STD-1399  Feedthrough 

1  Load  500W.  SOOV  20  amps,  1  TO  5  K  ohms  +/-  5% 

1  load  5W.  200V.  0.5A.  1  ->  100  K  ohms  +/-  025% 

Microwave  Signal  Interface 

Various  Coax  Connectors 

Cooling  Source  Inteifai,  i 

Unique  Per  Operational  TPS 

Heating  Source  Interface 

Unique  to  Operational  TPS 

Optical  Bench  and  Collimator 
Interface 

0.5  ->  10  Inches 

Air  Shutter,  Broad  Bandwidths 

Motion  Source  Interface 

Standard  INS  T^Ie  and  Unique  Interface  Per  OTPS 

Pneumatic  Interface 

Use:  <4  and  -6  “O”  Ring  Seal  Bulkhead  Unions  for  Pt  and  P, 
Respectively 

UUT  Structural  Support 

Interface 

Small  Shelf  For  UUT  or  Unique  Per  Operational  TPS 
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Tible  51.  Stimulus  Characteristics  of  CASS 


Stinnlas 

Characteristics 

Digital  Stored  Pattein 

Logic  Familes  •  TTL,  CMOS,  ECL,  DCL,  DTL,  RTL,  HTL,  MOS, 
CML 

336  I/O  Pins,  0  to  20/40  MBITS/Sec,  -5  ->  15  V 

Diagnostic  Probe  to  20  MBITS/Sec 

Analog  WsiveCcvins 

Signal  Gen.  @  0.01  ->  25  MHZ,  -5  ->  +5  Nfolts  Arbitrary  Wave  48 
HZ  ~>  200  MHZ,  TACAN,  ILS,  and  VOR  Modes,  Pulse  Genoator 

4  NS  ->  99.9  Sec,  Synchro  Generators 

Communication 

Wavefonns 

ModulatoR 

High  Accuracy  AM  (ILS  &  VOR),  TACAN,  Linear  Pulse,  FSK, 
MSK,  BPSK,  QPSK,  OQPSK(GPS),  Spread  Spectmm  10  MHZ  - 
>1.3  GHZ. 

Microwave  Signals, 
Modulation,  Broadband 
Noise 

Synthesizer  10  MHZ  —>  40  GHZ,  19  ->  -110  DBM,  Fast 

Switching  3  MHZ  -->  18.4  GHZ.  Noise  Source  10  MHZ  -->  26.5 
GHZ 

Eleciro-Opdc  Wavefonns, 
Sources 

1.064  uM  LASER 

Aperture  0.5  —>  5.0”,  5E-10  -->  5E-6  W/S(5CM,  Range  Gate  0.5  - 
>  10  KM,  FOV  20  ~>  500  mrad, 

1.064 +-0.01um 

Target  Collimator 

Selectable  Taigets  on  Wheel,  0.5  to  8  urad,  10  Inch  ^tenure,  7  ~> 

12  urn 

Visual  Targets.  03  to  8  urad,  1.1  urn 

Pneumatic  Functions 

0.6  ->  100  Inches  Hg  Absolute  Pressure,  0  ->  50,000  fi/min 

Table  52.  Measurement  Characteristics  of  CASS 


MeasurencBt 

Characteristics 

Digital  Response  Pattern 

Logic  Famites  -  TTL,  CMOS.  ECL,  DCL,  DTL,  RTL.  HTL, 
MOS.CML 

336  I/O  Pins.  0  TO  20/t40  MBITS/Sec.  -5  ->  15  V 

Diagnostic  Probe  to  20  MBITS/Sec 

Wavefnrn  Analysis 

Synchro  Resolver,  Multimeter  0  ~>  1000  Volts  (Probe  only 
capability  from  200  to  lOOOV),  Fieq.  0.(K)1  HZ  -->  200  MHZ, 
Time  4  NSEC  ->  15E3  Sec.,  Waveform  Digitizes  @  DC  -> 
500  MHZ.  8  BITS.  25uSEC  ~>  50  Sec. 

Demodulators 

High  Accuracy  AM  (ILS  &  VOR),  TACAN,  Linear  Pulse. 

FSK,  MSK.  BPSK.  QPSK.  OQPSK(GPS).  Spread  Spectrum 

10  MHZ ->  2  GHZ. 

Spectrum  Analysis.  >^lts. 

Power,  Phase 

Demodulation, 

Phase  Noise 

Power  -60  ->  +  2DBM  @  100  KHZ  ->  50  GHZ.  Peak  Power 

30  KHZ  26.5  GHZ.  Networic  Analysis  DC  ~>  26.5  GHZ; 

Spectrum  Analysis  -  1(X)  hz  -->  22  GHZ,  MOD  AM.  FM, 

Pulse.  MSK.  BPSK.  QPSK.  Demodulated  AM  and  FM. 

Electro-  Optical  Laser  Receiver 
Power/Frequency 

Modulation 

1.064um  LASER.  Aperture  0.5  ->  5.0  Iitch,  30  to  300M 

Joules,  Range  Gate  0.5  ~>  10  km.  FOV  20  -->  500  MRAD, 

1.064 -t-.  0.01  urn 

Optical  Detector 

Image  Analysis 

Tradter  30  to  300  M  Joules  @  1.064  urn. 

Aperture  OJ  ->  5.0  Inch,  FOV  30  DEG.,  PRF  8-20HZ. 

RS343,  RS170.  Raw  Video  to  40  MHZ 

Cunrent,  Voltage 

Power,  Noise 

Multimeter  0  ->  1000  Volts  (Probe  only  capability  from  200 
to  lOOOV),  Freq.  0.001  HZ  ->  200  MHZ,  Tune  4  NSec  -> 

15E3  Sec.,  Waveform  Digitizes  @  DC  ->  500  MHZ,  8  BITS, 
25USec  ~>  50  Sec. 

Pressure/Raie 

0.6  ->  1(K)  Inch  Hg  /  0  ~>  50,0(X)  It/min 

Tibie  53.  Oth^  Cliaracteristjcs  ot  CASS 


Owtrol 

•ad 

Switch 

ChanKteiiitic 

Test 

Equipacnt 

Opcmtiag 

Systaai 

Characteristic 

Power 

Programmable  Except 

Feedthrough  of  AC  Power 

115/200  V  3  PH  400  HZ 

Test  Language 

716-1985  C/ATLAS  With 
Extensions  (e.g.  EO),  Non 
ATLAS  Modulo  Ada.  C  or 
FORTRAN  (All  supported 

VMS  languages) 

Signal  Matrix 

Switch 

420  Pins.  Reliqr  Matrix  21@  1X4. 
35  @1X2.  DC ->1  MHZ 

192  Coax.  Relay  Matrix  27@ 

1X4. 9@  1X2.  DC  ->  16  GHZ 
76Power.5@  1X4.2®  1X2.6 
®  1X2 

Digital  Test 
Formats 

LASAR  V6  Source  A  Binary 

File  Format  for  L2(X)  Series 

Stiinulus 

Synchronization 

10  MHZ  Precision  Ref.,  and  TPS 

A  OTPS  ID.  Triggers.  Delays 

Complex 

Waveform 

716  C/ATLAS  Proposal  With 
Spread  Spectrum  Extensions. 

Measurement 

Synchronization 

10  MHZ  Precisian  Ref.,  and  TPS 

A  OTPS  Triggers.  Delays 

RealTinte 

Software 

Control 

Not  for  TPS  Developer 

Standard  IVotoools 
&  Buses 

MIL-STD-1553A/B.  1773. 

1397A.  EIA  RS232X:.  RS422-A. 
ARINC  AR-57A.  ARINC-429. 
IEEE  488. 802.3.  RS4S5 
Manchester.  (HSDB.  PODB  Not 
Populated  for  LRIP) 

Simultaneous 

Read 

Management 

Single  Thread  ATLAS  (With 
Specific  Exceptions,  e.g.  AWG 
Channels) 

Interface 

Characteristics 

Control 

ATLAS  Programmable  Control 

Maintenance 

Data/Services 

Interface 

Diagnostic  Results  for  CASS 
Logged,  UUT  Results  Logged 

UUT 

Synchronization 

Digital  Patton  Sync.  TPS  & 

OTPS  ID  Triggers 

Technical 

Manual 

Services 

lETM  Format  Based  On 
Commercial  Tools  A  Formats 

UUT  Mode  Control 
ID  Mode  Control 

All  Buses  And  All  Signal  Sources 

Self-Test 

Services 

Extensive  Power  Test  And 
Continuous  Monitoring 

Calitaatkm  Sources 

DC  0  ->  200  Volts  0.1  UV  ->  10 
UV 

AC  0.022  ->  220  VRMR  8  UV  - 
>220  UV 

DC  0 ->  2.2  A.  AC  0 ->  2.2  A. 
Resistance  1  ohm  -•>  19  Mohm 
@  lOma  Max 

Self- 

Calibration 

Services 

Automatic  Calibration,  One 

Year  Field  Use  Between 

External  Calibrations,  Based  on 
Built-In  Calibration  Standards 
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Tible54.  CASS  Test  Resources 


Misc.  Resources 


Common  Core 
Resources 
(Low  Frequency, 
Digital,  Bus) 

RF  Resources 

Digital  Multimeter 

Module 

Spectrum  Analyzer 
Assembly 

Frequency/Time  Interval 
Counter 

Power  Meter 

Waveform  Digitizer 

Microwave  ‘Transition 
Analyzer 

Pulse  Generator  Module 

40  GHZ  Synthesizer 

Arbitrary  Waveform 
Generator 

20  GHZ  High  Power 
Synthesizer 

Low  Wattage  Power  Load 

Fast  Switching 
Synthesizer  (Low 
Frequency) 

High  Wattage  Power  Load 

Fast  Switching 

Synthesizer 

(Microwave 

Extension) 

Switch  assemblies 

Modulation  Control 
Instrument 

Digital  test  unit 

RF  Interface/Coax 
Switch 

Commercial  Bus 

Intoface:  RS-232C;  RS- 
422A.  IEEE- 488. 1978; 
IEEE802.3-1985; 
ARINC429-10 

Spread  Spectrum 
Generator 

Military  Bus  Interface: 
MIL-STD.I553:  MIL- 
STD-1773;  MCAIR 

A3818 

Spread  Spectrum 
Modulators  And 
Demodulators 

Optional  Ancillary 
Resources 


Electro-Optical 

System 


UUT  D.C.  Power 
Supplies  -  Low 
Voltage  0-32V.O- 
100V.(M50V 


UUT  AC  PowCT 
SuppUesO-13SV 
Synchronized 


400  HZ  UUT  Power 
Feedthrough  and 
Monitor 


Inertial  Navigation 
System  -  AR-57A  Bus 
Interface 


Advanced 

Communication  Bus 
Interface  -  RS48S 
Manchester/Haipoon, 
HSDB(A12). 
FODB(A12) 


Military 

Communications  Bus 
Interface 


Pneumatic  Function 
Generator 
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8.1JZ  CASS  Introduction  Planning 

The  NAVAIR  PMA  260  (CASS)  Program  Office  maintains  a  current  schedule  of 
we^K>n  systems  and  subsystems  to  be  allocated  CASS  stations.  Table  SS  provides  a  list  of 
station  allocations  current  to  March  1992.  This  allocation  has  been  made  on  the  basis  of 
System  Synthesis  Model  (SSM)  analysis.  These  tables  illustrate  the  breadth  of  weapon 
system  requirements  attained  by  the  (TASS. 


Table  55.  Pit^rams  With  Planned  CASS  Station  Allocations 


AIWS 

AN/APG-73  Radar 
(F/A-18) 

HARM 

AMRAAM  Missile 

AN/APN-217(V)5 

Dealer 

HARPOON/SLAM  MissUe 

AN/AAS-33A 

JSOWMissUe 

AN/ALE-47  Chaff  Dispenser 

5400  COMP  F-14 

AN/ASN-139  CAINS  n 

AN/ALQ-126B  EW  (RF  SRAs) 

EA-6BADVCAP(RPG 

AAIP) 

MIDSLVT 

AN/ALQ-165  (ASPJ) 

EA-6B/ALQ-149 

PHOENIX 

AN/ALR-67ASR 

AN/ARC-210  Radio 

AN/USN-2V  SAHRS 

GPWS/HELO 

F-14DWRA/SRA 

SCS  (Standard  Compass  Sys.) 

GPWS/IRANS  (Tianqxnt) 

SH.60F/ALFS 

SH-60B  Upgrade  n 

F-14DIRSTS 

URC-107JTIDS 

AN/ZSD-1  AJARS 

AN/ALQ-156IDAP 

MAGR 

DAFCS  F-14 

AN/ALE-50  Decoy 

LPI  Altimeter 

Common  Recorder 

Table  56  shows  which  existing  NAVAIR  ATS  will  be  off-loaded  to  the  CASS 
stations.  Longer  term  planning  for  Navy  CASS  allocation  is  underway  for  (1)  Sparrow, 
Phoenix,  Sidewinder  Missiles,  (2)  F/A-18  E/F,  and  (3)  AFX. 


Table  56.  CASS  Station  OffiLoad  Programs 


AAM-60EOSTS 

AN/USM-247VAST 

S-3.F-I4 

AN/USM-470(V)2TMV 

F-14B 

AN/APM-446RSTS  F-18 

ASM-614  SH-60F 

AN/USM-470(V)  1  ATS 

F-18 

ASM-614  EA-6B 

AN/USM-403  HATS 

S-3 

AN/APS-137  S-3C  Radar 

EETS  (V)  1  AV-8B 

HTS  AV-8B 
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8.1  J  EFTE  General  Purpose  Station  Set 

IFTE  has  been  developed  and  manufactured  as  the  Army’s  standard  automatic  test 
equipment  for  commonality  across  field,  depot,  and  factory.  The  IFTE  has  demonstrated 
broad  based,  general  purpose  test  capability  as  the  following  analyses  indicate.  Yet,  an 
IFTE  Base  Shop  Test  Facility  (BSTF)  is  the  most  mobile,  general  purpose,  automatic  test 
equipment  ever  built  It  is  designed  to  be  transported  in  its  chemical,  biological,  and  nuclear 
protective  shelter.  The  IFTE  can  be  set  up  and  in  full  operational  mode  within  30  minutes. 

The  IFTE  Base  Shop  Test  Facility  is  composed  of  one  station  set  and  a  number  of 
standard  ancillary  units.  The  station  is  composed  of  low  frequency  analog,  digital, 
communication,  and  RF  capability.  The  EO  capability  is  provided  by  the  CASS  EO 
ancillary  unit  integrated  into  the  IFTE  with  an  IEEE  488  hardware/protocol  link  and  CASS 
EO  ATLAS  extensions  to  IFTE  ATLAS. 

In  addition,  the  commercial  equivalent  equipment  (CEE)  is  available  from 
Grumman  Coiporation  for  ATS  development,  depot,  and  factory  applications  not  requiring 
the  protective  shelter  capability  that  comes  standard  with  the  base  shop  test  facility.  The 
IFTE  BSTF,  the  CEE,  and  the  ATSE  (Army  Test  Support  Environment)  work  stations 
provide  the  full  spectrum  of  off-equipment  test  development,  maintenance,  and  operational 
use  capability  within  the  IFTE. 

The  key  attributes  for  each  of  the  principal  ATE  elements  (presented  in  Figure  1 ,  in 
Section  1.3)  were  listed  in  Table  49.  The  IFTE  capabilities  for  each  of  these  attributes  are 
summarized  in  Tables  57  to  60. 

Table  61  summarizes  the  test  instruments  and  resources  by  resource  category  type 
used  to  provide  the  key  stimulus  and  measurement  characteristics  of  the  IFTE  stations.  The 
test  resources  listed  as  optional  ancillary  resources  in  Table  45  do  not  affect  delivery 
schedule.  The  digital  and  analog  test  resources  are  listed  in  column  one.  Miscellaneous 
resources  are  used  in  one  or  more  configurations  and  optional  ancillary  resources  may  be 
used  in  any  configuration.  The  IFTE  station  set  and  optional  ancillary  units  provides  the 
most  mobile,  state  of  the  art,  general  purpose  automatic  testing  capability  in  DoD 
inventory. 


IWNe  57.  InterfiMe  Cteracteristks  of 


iBtarftKC 

Characteristics 

General  Puq)08e  Electrical 
Interface  lype 

Gold  Dot  3200.  Total  Quantity,  200  Volts/dot. 

200  Power  Paths,1400  Signal  Paths,  Allocated  as  Follows: 

130  (2  Dot)  Univttsal  @  1  amps  DC  to  40  MHZ  -3db 

144  (2  Dot)  ExL  Perf.  @  1  amp  DC  TO  lOOMHZ  •3db 

224  (2  Dot)  Digital  lO  Pins,  DC  •>  SO  MBIT/Sec 

32  (3  Dot)  Fonn  C  Relays 

145  (2  Dot)  Den^  Signal  Path 

1418  Signal  Dots  Unallocated 

11  Auxilliary  Interface  Panel  Connectors 

ProgranroaUe  Power 

Siqiplies  and  Programmable 
Loads 

Low  Voltage  12  @  17SOW/200V/72.8A 

Low  Voltage  Fixed  1  @  630W/28V/223A 

Reference  Voltage  8  @  4-10/-10V  /  lOOMA 

High  Voltage  -  O^V  @  0.7S  A 

Current  Source  —12  @  Ranging  0.7SA  ->  20.SA 

AC  Power  1  @  45  ->  5000  HZ,  4000W/270V/  10A/3PH 

8  Loads  @  750W.  Ranging  50V@30A  TO  250V 

10  Fixed  Resistance  SO  TO  1000  ohm 

Nfiaowave  Signal  Interface 

9  RF  OuQNits  -  IVpe  N  Precision  SS 

4  RF  IigNits  -  Type  N  Precision  SS 

Cooling  Source  Interface 

Unique  To  Operational  TPS 

Heating  Source  biterface 

Unique  To  Opermional  TPS 

Optical  Bench  And 

Collimator  Interface 

0.5" ->  5” 

Air  Shutter,  Broad  Bandwidths 

Motion  Source  Interface 

INS  Static  Load  100  LBS.  Rate  0-1000  DEG/Sec,  Acceleration 
3000  DEG/Sec/Sec.  Torque  0.8  Foot  LBS,  Option,  Other  by 

Unique  Interface  Per  OTPS 

Pneumatic  Interface 

Static  0.3 -36  Inch  Hg,  Total  03- 100  Inch  Hg.  Differential  0  to 
100 Inch Hg, Rate  120 Inch  Hg/ Min, Option 

UUr  Structural  Support 
Interface 

Small  Shelf  For  UUT  or  Unique  Per  Operational  TPS 

Table  58.  Stimulus  Characteristics  of  IFTE 


StiBUllUS 

Characteristics 

Digital  Stored  Pattem 

Logic  Famiks  -  TTL.  CMOS,  ECL,  EXTL,  DTL,  RTL,  HTL,  MOS, 
CML 

192  I/O  Pins.  0  to  20  MBIT/Sec  48  I/O  Pins  @  50  MBITS/SEc,  -10 
">  10  V,  Expansion  to  256  by  Channel  Cards/Max.  Channels  1024. 

32 10  Pins  @  10  MBITS/SEc  -30  ~>  30  V 

Current  Injection  Probe  (Bus  Source/Sink) 

Analog  Waveforms 

Signal  Gen.  @  DC  to  25  MHZ.  0  ->10  V  P-P  Arbitrary  Wave  20 

NS.  TACAN,  ILS.  and  VOR  Capable,  Pulse  Generator  40  NS  -> 

50  Sec,  Synchro/Resolver 

Communication 

Waveforms 

Modulators 

High  Accuracy  AM  (ILS  &.  VOR),  TACAN,  Linear  Pulse,  FSK, 
MSK,  BPSK,  QPSK,  OQPSK(GPS),  Spread  Spectrum  1  MHZ  -> 

2.3  GHZ. 

Microwave  Signals, 
Modulation,  Broadband 
Noise 

Synthesizer  10  KHZ  ~>22  GHZ,  38  — >  -120  DBM,  Fast  Switching 
1.0  MHZ  TO  2.3  GHZ,  Broadband  Noise  Source  -  lOMHZ  -  26.5 
GHZ -Option 

Electro-Optic 

Waveforms,  Sources 

RS343,  RS170,  Raw  Video, 

12  Tuget  Simulator  (IF,  RF,  Video)  Barker.  PN,  Doppler  -  Option 
1.064  urn  LASER  Aperture  0.5  ->  5.0  Inch.  0.5E-10  ->  0.5E-6  W/ 
SQCM,  Range  Gate  0.25  ~>  10km,FOV20~>500MRAD.  1.064 
+-  0.01  um 

Target  Collimator 

Selectable  Targets  on  Wheel  7  to  67  DEG  C,  10’  Aperture,  1  —>11 
um 

Vsual  Targets  1’  by  1”  Max  0.6  ->  1.1  um 

Pneumatic  Functions 

Static  03  -  36  Inch  Hg,  Total  0.3  - 100  Inch  Hg,  Differential  0  to 

100  Inch  Hg,  Rate  120  Inch  Hg  /  Min,  Option 

IkMe  59.  Measurement  Characteristics  of  IFTE 


Measurmcrt 

Characteristics 

Digital  Response  Putem 

Logic  Familes  -  TTL,  CMOS,  ECL,  DCL,  DTL,  RTL,  HTL.  MOS, 
CML 

192  I/O  Pins,  0  to  20  MBIT/Sec  48  I/O  Pins  @  SO  MBITS/SEc.  - 
10  ~>  10  V,  Expansion  to  2S6  by  Channel  Cards/max.  Channels 
1024 

32 10  Pins  @  10  MBITS/Sec  -30  ~>  30  V 

Diagnostic  PrtA)e  to  SO  MBITS/Sec 

Wavefonn  Analysis 

Angle  Position  Indicator,  Multimeter  0  -->  200  V,  Req.  0.0001  HZ 
~>  100  MHZ,  Hme  100  NSec  ~>  1(£3  Sec.,  Waveform  Digitizers 
@  DC  ->  so  MHZ,12  BITS,  10  NSec  ->42.9  Sec. 

Demodulators 

High  Accuracy  AM  (ILS  &  VOR),  TACAN,  Linear  Pulse,  FSK, 
MSK,  BPSK.  QPSK,  OQPSK(GPS),  Spread  Spectrum  1  MHZ  -> 
2.3  GHZ, 

Spectrum  Analysis,  Volts, 
Power,  Phase 

Demodulation, 

Phase  Noise 

Spectrum  Analyzer  100  HZ  ->  22  GHZ  440  DBM  ~>  -100  DBM. 
Power  -70  ~>  4  40  DBM  @  8  MHZ  ~>22  GHZ.  Power  Via 
Millivolt  M^  to  1.2  GHZ,  Prog.  Atterttuator  0  ->  100  DB  to 

440  DBM,  Network  Analysis  4S  KHZ  ~>  26.S  GHZ,  Phase  Noise 
lOMHZ  ->  18  GHZ  -  Option 

Electro- Optical  Laser 
Receiver  Power/Fiequency 
Modulation 

1.064um  LASER,  Aperture  0.5  ~>  5.0”,  5E-10  ~>  5E-6  W/ 

SQCM,  Range  Gate  025  ->  10  km.  FOV  20  ~>  500  MRAD, 
1.0644-0.01  urn 

Optical  DeteclKH’ 

Image  Analysis 

Ttacker  5E-10  ->  5E-6  W/SQCM  @  1.064  um. 

LASER  Transmitter  Aperture  0.5”  to  5.0”,  FOV  30  DEG.,  PRF  8- 
20HZ,  Divergence, 

RS343,  RS170.  Raw  Video  to  40  MHZ 

Current,  Nfeltage 

Power,  Noise 

Multimeier  0  ->  200  Volts,  Freq.  DC  ->  100  MHZ,  Time  100 
NSec  ->  10E3  Sec.,  Waveform  Digitizers  @  DC  ->  50  MHZ,  12 
BITS,  lONSec  ->42.9  Sec. 

Pressuie/Rate 

Ancillary  Equipment 
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Hable  60.  Other  Characteristics  of  IFTE 


Coatrol 

and 

Switch 

Characteristic 

Test 

Equipiaent 

Operatiag 

System 

Characteristic 

Power 

Programmable  Exeqx  As 

Noted  Under  Interface 

Test  Language 

ATLAS.  Non  ATLAS 

Modules 

Signal  Matrix 
Switch 

10  Signal  Buses,  130 

Universal  10  (40MHZ  -3  DB), 
48  Instrument  Ports  -  Intmnai 
Switching,  144  Extended 
Performance  Direct  Connect 
to  48  Instruments  (100  MHZ  • 

3  DB),  32  Form-c  Utility 

Relays 

Digital  Test 
Formats 

LASARV6C^le 

MATE-STD-SDF 

All  Digital  Tests  Developed 
Using  ATLAS  Siatments 

Stimulus 

Synchronization 

10  MHZ  Precision  Ref., 

Systtun  IViggers/  Delays,  and 
UUT 

Complex 

Waveform 

None  Direct,  Must  Use 
Existing  1985  C/ATLAS 
and  AFG 

Measurement 

Synchronization 

10  MHZ  Precision  Ref., 
Systems  lYiggers/Delays,  and 
UUT 

Real  Time 
Software 

Control 

In  “Ada”  and  “C"  Modules 

Standard 

lYofDColsand 

Buses 

ME..STD-  1553  A/B,MIL- 
STD-1773, 1397A,EIA 
RS232-C,RS422-A,AR- 
57A,  IEEE  488, 8013,  RS485 

Simultaneous 

Thread 

Manag^nent 

Single  Threaded  ATLAS 

Int^ace 

Characteristics 

Control 

ATLAS  Programmable 

Control 

Maintenance 

Data/Services 

Interface 

Logging  UUT  Test  Results 

Logging  Mainteiuuice  Test 
Results 

UUT 

Synchronization 

Digital  Pattern  Sync,  TPS  & 
OTPS  ID  Triggers 

Technical 

Manual 

Services 

Cals  Compaiable  lETM 

UUT  Mode 

Control 

ID  Mode  Control 

All  Buses  And  All  Signal 
Sources 

Self-Test 

Services 

Extensive  Power  On  Self- 
Test  and  Continuous  BIT 
Monitoring,  Self  Test  ID. 
ATLAS  Initiated  Self  Test 

Calibration 

Sources 

DC  0.09, 0.9, 9.0, 18.0  V +- 
0.008%/YR 

AC  1, 10  VRMS  +-  0.06%/YR 
Resistance  100  ohm  ->  1 

Mohm  +-  0.0055% 

Prog  Load  Currait  Sense  Res. 
STDS  0.01  TO  0.06  ohm +- 
.55% 

Thermal  Transfer  Standard 

Self  Calibration 
Services 

Automatic  (Calibration  With 
Self-Alignment  ID 
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'Hible  <1.  IFTE  Itet  Resources 


Low  Freqimcy  Aoalog 
aod  Digital, 


Digital  Multimeter 
Module 


Fiequency/Time  Interval 
Counter 


Wavefom  Digitizer 


Pulse  Generator  Module 
as  a  Function  of  AWG 


Arbitrary  Function 
Generattv 


Low  wattage  power  load 


RF  Resources 


Spectrum  Analyzer  to 
22GHZ 


Power  Meter  to  22 
GHZ 


Millivolt  Meter  12 
GHZ 


Synthesizer  to  22  GHZ 


RF  Inteiface/Coax 
Switch 


Fast  Switching 
Synthesizer  to  13 
GHZ 


MISC.  RESOURCES 


Synchro/Resolver/ 
Indicator  &  Simulate’ 


UUT  D.C.  Power 
Supplies  -  Low 
Voltage  to  36v.  High 
Voltage  to  600V, 


UUT  AC  Power 
Supplies  0-270V  3  ph. 


High  Power  Load 
(3000W) 


Precision  Reference 
Source  (+/-10V) 


Digital  Test  Unit  VTith  Pseudmandom 
Voltage  and  Current  Probe  Sequencer 


Video  Analyzer  and 
Simulator 


Switch  Assemblies 


Commercial  Bus 
Interface:  RS-232C;  RS- 
422A;  RS-423;  RS-485. 
IEEE  488,1978: 
IEEE8023.198S;  VME 


Military  Bus  Interface: 
MIL-STD-1553A/B, 
MIL-STD-1773,M1L- 
STD-1397A,  AR-57A 


Spread  Spectrum 
Modulate  and 
Demodulators 


OPTIONAL 

ANCILLARY 

RESOURCES 


ElectroOptical 

System 


Broadband  noise 


Sampling  Signal 
Analyzer  to  26  GHZ 


Noise  Figure 
Measurement 


VXI  Chassis 


Low  Phase  Noise  RF 
GeneiatOT 


Canier  Noise  Test  Set 


Speenum  Analyzer  to 
22  GHZ,  FFT  Under  1 
KHZ  Measurements 


8.1.4  IFTE  Laydown  Planning 

The  introduction  of  the  IFTE  is  referred  to  as  the  IFTE  laydown  by  the  Army.  The 
funded  and  near  term  projected  (FY93  to  FY95)  laydown  of  IFTE  BSTS  are  summarized 
in  Table  62.  Two  principal  Army  ATS  types,  the  EQUATE  and  Hawk  High  Frequency 
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Console,  will  be  replaced  by  IFTE.  The  data  in  this  table  was  provided  by  the  Army  during 
the  TIMs,  and  is  subject  to  changes  resulting  from  DoD  budget  revisions.  Information  from 
the  IFTE  Program  Office  in  of  March  1993  implied  that  current  budget  trends  indicate  that 
revised  quantities  may  be  closer  to  200  total. 


liable  62.  Planned  Laydown  of  239  IFTE  ATS  (FY93-FY95) 


7  each  Avenger 

S3  each  Hawk 

50  each  Hawk  Phase  II 

5  each  ASAS 

12  each  Singars 

31  each  MLRS 

22  each  Paladin 

1  each  Javelin 

41  each  EQUATE 

17  ea  (Other  Support) 

Possible  upgrades  UAV,  Apache,  Ml,  and  Bradley 

8.2  WEAPON  SYSTEM  TEST  REQUIREMENTS  ANALYSES 

Cross-weapon  and  cross-Service  ATS  requirements  analyses  were  conducted  in 
conjunction  with  this  study.  These  analyses  demonstrated  that  CASS  and  IFTE  exceed 
most  technical  test  requirements.  More  importantly,  since  CASS  and  EFTH  use  flexible 
industry  standard  architectures  that  promote  modularity  and  partitioning  (e.g.,  MMS, 
VME,  Ethernet),  they  may  be  updated  relatively  easily.  As  significant  performance 
enhancing  technologies  become  available,  the  CASS  and  IFTE  families  can  incorporate  the 
improvements  without  disturbing  basic  architectures. 

The  cross-weapon/Service  review  compared  difficult  to  test  weapon  system  items 
against  the  capability  of  selected  automatic  test  equipment  to  meet  related  technical  testing 
requirements.  F/A-18  subsystems  were  compared  witfi  IFTE,  F-22  subsystems  (projected) 
were  compared  with  CASS,  and  Apache  subsystems  were  compared  with  the  F-16  lAIS 
ATS.  The  Army  and  Navy  were  asked  to  select  assemblies  that  represent  a  challenge  for 
other  ATS.  Since  actual  test  requirements  for  an  F-22  system  have  not  been  determined  yet, 
the  F-22  Common  Automatic  Test  System  (CATS)  preferred  equipment  list  was  used  as  a 
comparison  baseline.  Three  F/A-18  weapon  replaceable  assemblies  (WRAs)  were 
evaluated  against  the  IFTE  Base  Shop  Test  Facility.  Finally,  in  order  to  gain  insight  into 
the  capability  of  existing  ATS  to  support  cross-Service  testing  requirements,  three  Apache 
line  replaceable  units  (LRUs)  were  evaluated  against  the  F-16  Improved  Avionics 
Intermediate  Station  (lAIS). 


Comparisons  results  are  presented  in  Table  63.  Other  comparisons  of  the  F- 1 6  lAlS, 
and  CASS  and  IFTE  showed  these  ATS  family  candidates  have  more  extensive 
capabilities.  For  example,  CASS  and  IFTE  possess  both  LRU  and  SRU  testing  capabilities. 


while  the  lAlS  focuses  principally  on  LRU  testing.  This  taUe  also  addresses  the  exceptions 
and  what  the  impact  of  these  exceptions  might  be.  The  “Exceptions”  in  Table  63  are 
grouped  as  none,  minor,  nKxlest,  and  major. 

Table  63.  Cross-Service  ATS  Review  Results 


SYSTEMS 

ITEMS  COMPARED 

EXCEPTIONS 

Radar  Set  Receiver-Extension 

None  -Full  capability  exists 

Radar  larget  Data  Processor 

F-18WRAsto 

IFTE 

FLOt  Receiver  A  Optics  Stabilizer 

Minor  -Stimulus  and  measurement 
requirements  within  capabilities 

-  IR/EO  testing  needs  HW  fixturing 

augmenUttion 

-  Optics  stabilizer  needs  HW 

augmentation  for  vibration  test  aand 
and  motor-powered  mechanical 
.interface 

VXI  Instnunents 

None  •  Comparable  capabilities 

F-22  CATS 
Preferred 
Equipment  List 
to  CASS 

RF  Equipment 

None  •  Technical  capabilities 
comparable.  CATS  l^e  rack  &  stack 
equipment,  CASS  downsized  MMS 
based 

EO  Equipment 

None  -  Ciqnbilities:  CATS  limited, 
CASS  broad  based 

R'lVS  Electronic  Unit  Assembly 

None  -  Full  capabUity  exists 

Apache  LRUs 
to 

PVNS  Night  Sensor  Assembly 

None  -  Stimulus  and  measurement 
requirements  exist,  holding  fixture 
augmentation  needed 

r^lO  lAlS 

Day  Senjor  Assembly 

None  -  TPS  augmentation  of  a 
transformer  (on  qualified  parts  list- 
QPL)  needed 

TADS  Night  Sensor  Assembly 

None  -  Temperature  probe,  voltage 
divider  and  variable  temperature  target 
source  required  for  TPS  iniettace 
adapter 

•  A  minor  exception  is  defined  as  a  change  or  extension  of  test  operation  software  and 
the  addition  of  a  few  test  resources  not  necessarily  affecting  the  overall  ATE  foot¬ 
print 
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•  A  modest  exception  is  defined  as  the  addition  of  a  new  chassis  resource  which  may 
lead  to  an  extension  of  the  ATE  foo^rint  after  a  few  are  implemented. 

•  A  major  exception  is  defined  as  a  new  configuration  with  an  enlarged  footprint  by 
20%  or  more. 

Few  minor  and  no  modest  nor  major  exceptions  were  identified. 

Additional  comparisons  of  DoD  tester  capabilities  are  presented  in  Appendix  D. 
Based  on  the  analyses  from  these  comparisons,  only  minor  tester  capability  technical 
improvements  to  ATS  families  will  be  required  over  the  next  decade. 

8.2.1  A*12  Test  Requirements 

Thedevelopmcntofthe  A- 12  test  performance  envelope  for  a  CASS  IOC  1993  was 
progressing  effectively  up  until  the  time  the  program  was  cancelled.  In  one  study  [CASS 
1992]  done  for  NAVAIR  552,  it  was  determined  that  the  Navy’s  replacement  for  the  A6 
aircraft,  the  A- 12  (now  the  AFX)  to  be  fielded  perhaps  before  the  year  2005  would  be  fully 
tested  by  the  CASS  test  suite  with  the  addition  of  three  new  cards.  With  respect  to  the 
summary  of  CASS  capabilities  presented  earlier,  these  resources  fall  under  the  Control  and 
Switch  characteristics  listed  in  Table  53  under  Standard  Protocols  &  Buses. 

Because  these  cards  represent  a  small  set  of  the  interfaces  required  to  test,  these 
cards  represent  a  less  than  5%  change  in  the  number  of  unique  test  resources  available 
within  the  CASS.  Test  resources  represent  approximately  60%  t^e  total  value  of  a 
complete  CASS  station  set  Therefore,  only  minor  changes  (less  than  3%  impact)  are 
required  to  accommodate  the  most  advanced  avionics  technology  known. 

8.2.2  AFX  Test  Requirements 

During  the  ATS  study  questions  arose  concerning  CASS  compatibility  with 
JIAWG  modules  and  components  anticipated  as  elements  of  F-22  LRMs,  LRUs,  and  SRUs 
(therefore,  comments  apply  equally  to  F-22).  The  Navy  performed  [JIAWG  1992] 

...  an  analysis  of  approximately  forty  specirications  which  establish  the 
Common  Avionics  Baseline  [CAB  in  Revision  3, 2  Dec  1991]  for  the  AX. 

The  areas  investigated  included  JIAWG  architecture,  module  and  processor 
standards,  system  and  module  bus  specifications,  input/ouq>ut  and  BIT 
specifications,  test  software  specifications.  Software  Engineering  Environ¬ 
ments,  Reliability  and  Maintainability  and  Integrated  Logistic  NAVAIR ’s 
current  ASTEC,  Testing  2000  and  ABET  efforts. 
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Table  64  d^ines  the  major  componoits  of  interest  to  diis  analysis.  Additional 
q)ecificati(ms  under  development  and  of  interest  to  this  analysis  include: 

a.  High  Bandwidth  Interface  Module 

b.  Signal  Interface  Module 

c.  \^deo  Data  Interface  Module 

d.  Signal  Data  Distribution  Network 

e.  Video  Data  Distribution  Network 


Table  64.  JIAWG  Specifications  For  CASS  Analysis 


JUWG  SPEC 

PACKAGING 

DESCRIPTION 

J87-01 

SUBSYSTEM 

Advanced  Avionics  Aichitec&ire  (A3) 
Standard 

J90-CNI-01 

SUBSYSTEM 

Integrated  CNI  System  Standard 

J89-SP-01 

SUBSYSTEM 

Signal  Processor,  Common  Avionics 
Processor 

J88-M1D 

LRM 

Data  Processor,  Common  Avionics 
Processor  (CAP-16) 

J88*M2D 

LRM 

Data  Processor,  Common  Avionics 
ProcessOT  (CAP-32) 

J88-MSD 

LRM 

High  Speed  Data  Bus  (HSDB) 

Interface  Module) 

J88-M6D 

LRM 

1553  Mult^lex  Data  Bus  Interface 
Module  Specification 

J89-N1A 

SUBSYSTEM  BUS 

JIAWG  ParaUel  Inter-module  Bus 
(JPI-Bus) 

J89-N1B 

SUBSYSTEM  BUS 

JIAWG  Test  and  Maintenance  Bus 
(JPI-Bus) 

J89-N1C 

SUBSYSTEM  BUS 

Utilities  Signals  Specification 

J89-N1F 

SUBSYSTEM  BUS 

User  Console  Interface  Specification 

J89-S10 

SUBSYSTEM 

MISCELLANEOUS 

Module  Initialization,  Test  and 

Software  Interface  Specifications 

J88-G6 

SUBSYSTEM 

MISCELLANEOUS 

Integrated  Logistic  Support  Standard 

J88-G3 

SUBSYSTEM 

MISCELLANEOUS 

Reliability  and  Maintainability 

Standard 
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Preliminary  information  indicates  that  these  are  point  to  point  buses  operating  in 
excess  of  200Mhz. 


The  general  conclusion  to  be  drawn  from  the  specifications  reviewed  is  that 
CASS  will  still  be  required  for  all  of  the  non-JIAWG  components  in  the 
weapon  system  (RF  transmitters  and  receivers,  electro-optics,  displays,  etc.) 

With  regard  to  the  JIAWG  elements,  the  Navy  reports: 

The  voltage,  logic  levels,  impedance,  speed  of  the  module  interfaces,  pin 
count  and  other  parametric  of  the  JPl,  and  JTM  buses  are  well  within  CASS 
capability. 

The  Navy  report  concludes: 


1.  CASS  has  the  inherent  capability  to  test  JIAWG  modules  as  currently  de¬ 
fined  by  CAB  ID  revision  3  of  December  1991,  but  some  effort  will  be  re¬ 
quired  to  address  the  protocol  of  the  JPI  and  JTM  high  speed  data  buses. 

In  fact,  the  CASS  Digital  Test  Unit  (DTU)  with  proper  programming  can  test  JPI 
and  JTM  (nomenclature  for  two  high-speed  data  buses).  However,  for  efficiency,  when 
functional  stimulation  is  desired,  it  would  be  useful  to  have  a  new  stimulus  and 
measurement  standard  bus  module  as  an  ancillary  unit  (e.g.,  the  ACBI).  The  Navy  report 
goes  on  to  state: 


2.  Additional  JIAWG  components  will  be  developed  and  will  have  to  be  ad¬ 
dressed.  These  include  signal  data  and  video  data  interfaces,  image  process¬ 
ing,  graphics  processing  and  EW  processing. 

JIAWG  requires  that  module  BIT  software  interfaces  be  written  in  Ada.  This  Ada 
code  may  be  used  to  support  test  vector  application  generation  either  through  software  tools 
used  to  generate  a  LAS  AR  stored  stimulus/response  database  or  dirough  direct  execution 
of  a  non- ATLAS  module.  A  specialized  into^active  Ada  BIT  interface  test  module  as  an 
extension  to  the  CASS  stimulus  and  measurement  modules  may  also  be  required. 


The  examination  of  the  AX  ATS  requirements  compatibility  with  CASS  indicated 
that  there  are  no  known  exceptions  that  would  prevent  the  use  of  CASS.  In  fact,  CASS  had 
more  capability  than  the  other  testers  reviewed  and  appeared  to  be  the  best  ATS  match  to 
meet  the  anticipated  AX  test  requirements.  Further,  testing  areas  yet  to  be  determined 
represent  a  small  fraction  of  the  ATS  requirements,  and  most  fall  in  the  area  of  standard  bus 
protocols  that  will  ultimately  need  to  be  addressed  for  multiple  systems  applications. 
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8J13  F-22  llBst  Requiremoits 

The  F-22  CATS  ATS  will  permit  easy  configuration  of  separate  instrument  on  a 
card  to  meet  dedicated  factory  (and  possible  depot)  testing  requirements  for  digital  and  low 
frequency  analog.  CATS  is  not  well  suited  for  microwave  or  RF  since  it  currently  does  not 
include  the  MMS  architecture.  This  is  a  probable  future  addition  to  CATS,  without  which 
some  testing  capabilities  may  not  be  attainable. 

The  integration  of  general  purpose  robust  testing  capabilities  (equivalent  to  CASS 
or  IFTE)  will  require  additional  ATS  design.  This  design  is  currently  unfunded  and 
unplanned.  Furthermore,  reuse  of  test  programs  from  full  sized  GPIB  instruments  suites 
like  the  (TATS  to  downsized  Mevel  test  equipment  like  the  CASS  will  be  difficult.  In 
general,  instrument  performance  will  be  different  in  downsized  MMS  or  future  VXI  RF 
instruments. 

The  I-level  or  depot  level  test  requirements  for  the  F-22  were  not  available  for 
study.  However,  die  Air  Force  did  provide  a  preferred  equipment  list  for  factory  test 
equipment  [CATS  1992].  Interesting  properties  of  this  equipment  list  are  depicted  in  Table 
65.  Most  notable  is  the  lack  of  MMS  RF  and  microwave  downsized  packaging.  During  the 
ATS  study,  the  instrument  vendors  indicated  that  commercial  enterprises  would  not  pay  the 
extra  cost  for  downsized  test  equipment  The  contractors  have  chosen  modified  VXI  for 
relatively  inexpensive,  simple  general  purpose  test  functions  and  full  sized  GPIB 
packaging  for  microwave  and  RF 

From  a  different  perspective,  the  AX  WRAs  and  SRAs  are  expected  to  be  based  on 
the  same  technology  as  the  F-22  aircraft  Thoefore,  based  on  analysis  summarized  earlier 
in  this  section  of  the  report  CASS  will  provide  excellent  test  coverage  for  the  known  F-22 
requirements. 


Table  65.  F>22  CATS  Preferred  Equipment 


Architecture 

Number  of  lustrumeuts 

Comments 

Modified  VXI 

37 

Provided  by  IS  instniment 
manufactures 

Modified  VXI 

9 

All  instruments  in  this  class 
are  power  or  switch  modules 

GPIB 

26 

IEEE-488  based  microwave 
and  RF  instruments 

IEEE  488  RACK  &  STACK 

30 

OEM  components  that  are 
accessories  to  GPIB 
instruments 

TBD 

1 

High  voltage  power  suf^ly 

Totals 

103 

Test  resource  items 

8.2.4  F-15  Test  Requirements 


Recent  studies  by  the  Air  Force,  Army,  and  Navy  have  concluded  that  CASS  and 
IFTE  meet  the  technical  testing  requirements  of  the  Air  Force’s  new  requirement  for  a 
downsized  tester  with  only  minor  enhancements.  Due  to  differing  user  mobility 
requirements,  the  IFTE  BSTF  appears  most  compatible  with  stated  requirements.  The  Air 
Force  stated  its  principal  concern  rested  with  IFTE  schedule  risks  that  had  the  potential  of 
impacting  overall  need.  However,  subsequent  analysis  by  the  Army  observed  that  IFTE 
would  actually  lower  overall  program  risk  and  could  be  available  far  earlier  than  a  new  F- 
15  Downsized  Test  Set  (DSTS)  acquisition. 


Test  requirements  that  were  not  met  widi  the  current  Army  configuration  of  the 
BSTF  were  phase  noise  measurement  for  ground  tracking  radar,  possible  higher  accuracy 
RF  frequency  measurement,  and  added  procedure  libraries  invoked  via  the  “perform” 
statements  for  avionics  waveforms.  Analyses  showed  that  these  additional  capabilities 
could  be  added  with  minor  impact  to  schedule  and  cost. 


8.2.5  F/A>18  Test  Requirements 

NAV AIR-552  selected  a  number  of  F/A-18  WRA  test  requirements  documents 
(TRD)  which  would  challenge  the  BSTF  ATS  test  capabilities  envelope.  The  Army  IFTE 
BSTF  TPS  analysis  team  evaluated  these  TRDs  compared  to  IFTE  BSTF  capabilities  [F/ 
A- 18  1992]  and  returned  the  following  result: 
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A  review  of  the  TRDs  for  FIA-18  Radar  Set  Receiver-Exciter  and  the 
Radar  Target  Data  Processor  was  made  to  determine  if  they  are  testable  on 
the  BSTF  or  the  CEE.  Each  TRD  was  analyzed  to  determine  power,  stimu¬ 
lus,  and  measurenient  requirements.  Table  A  [see  Appendix. . .  ATS  Study 
Analyses]  is  a  summary  of  this  analysis.  The  test  envelope  of  both  the  Re¬ 
ceiver-Exciter  and  the  Radar  Target  Data  Processor  are  within  the  capabili¬ 
ties  of  the  CEE  and  BSTF. . . 

A  review  of  the  Test  Requirements  Documents  (TRD)  for  the  FIA-18  FUR 
Infrared  Receiver  and  Optics  Stabilizer  was  conducted  to  determine  if  they 
were  testable  using  the  IFTE.  While  the  ITU^s  accurately  defined  AC  and 
DC  Power  Requirements  in  a  tabular  form,  stimulus  and  measurement  re¬ 
quirements  were  not  clearly  defined.  A  review  of  the  documentation  re¬ 
vealed  no  stimulus  or  measurement  requirement  that  is  not  within  the  IFTE. 

The  BSTF  Electro-Optic  B  (EOB)  would  be  the  basis  for  all  IR/EO  testing 
augmented  by  special  (TPS)  fixturing  as  required. . . 

The  Infrared  Receiver  does  not  require  the  elaborate  mechanical  fixturing 
but  it  does  have  optical  test  requireiiKnts  that  has  the  capability  of  generat¬ 
ing  an  infrared  target  from  0  to  99  degrees  (Celsius  with  a  resolution  of  0.01 
degree  Celsius.  The  capability  of  die  Navy  provided  Electro-optical  work 
bench  (verify  referring  to  the  CASS  EO)  is  +30  to  -10  (degrees  Olsius) 
from  ambient  temperature.  Additional  target  generation  would  be  provided 
using  a  (TPS)  future. . . 

In  summary,  the  weapon  system  capability  and  requirements  comparisons  indicate 
that  the  BSTF  ATE  is  capable  by  virtue  of  its  lest  capabilities  envelope  to  test  the  F/A-18 
WRAs. 

8.2.6  APACHE  (AH-64)  Test  Requirements 

An  analysis  of  the  F16  AIS  and  lAIS  was  performed  for  capability  to  test  Army 
aviation  electronics.  The  Army  again  selected  the  most  difficult  LRUs  for  analysis.  The 
lAIS  turned  out  to  be  a  very  capable  ATS,  with  almost  all  requirements  covered  with  the 
exception  of  multiple  targets  for  FLIR  testing. 

Five  major  LRUs  were  compared.  One  Apache  TADS/PNVS  Targeting  and  Night- 
Vision  subsystem  and  four  LRUs  from  the  Apache  Targeting  and  Night- Vision  System 
were  selected  by  the  Army  for  the  review  to  determine  if  the  Improved  F-16  AIS  (lAIS) 
could  meet  their  testing  requirements.  General  Dynamics  Electronics  Division  reported  the 
results  [TADS/PNVS  1992]  [APACHE/F-16AIS  1992]. 

The  test  parameter  matrix  was  constructed  utilizing  appendix  D  of  the 
TRDs.  Tolerance  assessments  for  both  depot  and  AVIM  were  compared  to 
lAIS  measurement  capabilities.  Each  test  was  reviewed  for  required  stimu- 
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lus  and  results  compared  to  lAIS  stimulus  capabilities.  Comparison  to  AIS 
capabilities  assumed  that  the  lAIS  Optical  Test  Bench  (0TB)  would  replace 
the  AIS  0TB. 

The  stimulus  and  measurement  capability  for  all  tests  of  the  PNVS  Electron¬ 
ics  Unit  Assembly  presently  exists  within  the  lAIS. 

The  stimulus  and  measurement  capability  for  all  tests  of  the  PVNS  Night 
Sensor  Assembly  presently  exists  except  for  the  measurement  of  a  mechan¬ 
ical  angle  of  22  +-2  degrees.  This  capability  would  be  integrated  into  the 
TPS  holding  fixture. 

The  stimulus  and  measurement  capability  for  all  tests  of  the  Day  Sensor  As¬ 
sembly  presently  exists  except  for  the  stimulus  of  6.6  VRFMS,  5  AMP,  400 
Hz.  This  capability  would  be  integrated  into  the  TPS  electronics  hardware 
as  a  transform^',  QPL  M27/335-02. 

The  stimulus  and  measurement  capability  for  all  of  the  tests  of  the  TADS 
Night  Sensor  Assembly  presently  exists  except  for:  (1)  the  measurement  of 
temperature  in  degrees  Celsius,  (2)  die  stimulus  of  a  ninth  E)C  power  supply 
which  would  be  added  to  the  TPS  intoface  test  adaptor,  and  (3)  a  variable 
temperature  target  source  for  infrared  sensor  testing  producing  absolute  or 
differential  temperature  control  for  one  or  more  targets. 

In  summary,  the  lAIS  provided  excellent  test  capability  coverage  with  the 
exception  of  electro-optic  target  stimulus  attribute  for  FLIR  testing  targets  which  are  not 
part  of  the  lAIS  current  mission  requirements. 

DOD  ATS  COMMON  R&D  REQUIREMENTS 

Three  specific  investment  areas  have  been  identifred  for  common  R&D  necessary 
to  maintain  a  DoD  family  of  capable  ATS.  These  areas  include  (1)  ATS  family 
convergence,  (2)  next  generation  test  software  environment,  and  (3)  ATS  family 
modernization.  This  common  R&D  discussion  assumes  that  each  specific  ATS  family 
member  will  have  its  own  R&D  as  well  as  P^I  maturation  or  modernization  budgets. 

The  common  R&D  cuts  across  Service  and  weapon  system  requirements  to  identify 
and  develop  the  following: 

a.  Methods,  tools,  military  and  commercial  standards  to  improve  specific  ATS 
concept  and  design  development  that  will  support  interoperability  of  existing 
and  future  family  members. 

b.  Improved  next  generation  test  software  environment. 

c.  New  test  functions  that  address  fundamental,  new  test  requirements 
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Table  66  identifies  several  inqxxtant  aq)ects  of  each  of  the  three  principal 
investment  areas.  These  inqxntant  aq)ects  are  presented  as  three  separate  and  distinct  lists 
under  each  of  the  investment  areas. 

Table  66.  Common  ATS  R&D 


8.4  DOD  ATS  FAMILY  IMPLEMENTATION 

Designated  DoD  ATS  should  continue  to  be  matured  as  they  are  fielded  to  obtain 
the  full  system  performance.  It  is  anticipated  that  these  systems  will  require  test 
requirements  extensions  throughout  their  lO-to-20  year  life  cycle.  Designated  DoD  ATS 
architectures  were  designed  to  be  flexible  and  to  accommodate  technology  improvements. 
These  in^ovements  must  require  upward  compatibility  so  that  TPSs  developed  for  the 
previously  fielded  configurations  will  run  on  new  improved  configurations  with  the  same 
test  resources. 

Limited  test  requirements  cr^ab  technical  improvements  to  ATS  family 
members  will  be  requited  over  the  nex  ^ade.  New  test  requirements  identified  in 
Technical  Interchange  Meetings  (TIMs)  and  subsequent  meetings  with  the  Navy  Program 
Office  and  die  Army  Program  Office  have  identified  that  these  potential  test  requirements 
for  common  use  across  multiple  systems  and  Services  may  be  integrated  into  die  designated 
DoD  ATS  platforms. 

The  test  performance  requirements  envelope  analyses  are  performed  with  the  aid  of 
an  automated  tool  developed  during  the  CASS  program.  This  tool,  the  System  Synthesis 
Model  (SSM),  is  critical  to  defining  brojul  complex  test  performance  envelopes  to  be 
compared  with  new  weapons  test  requirements.  As  new  test  requirements  are  identified. 


these  requirements  must  be  evaluated  with  SSM  models  to  determine  if  test  requirements 
lay  outside  of  the  performance  envelope  of  existing  family  members.  If  so,  then 
enhancements  of  capabilities  of  existing  family  members  must  be  identified  and 
implemented.  Areas  of  possible  extended  test  requirements  were  identified  and  presented 
in  the  tables  of  Part  H,  Summary,  of  this  paper.  Section  2.3.1,  CASS  and  IFTE  Technical 
Capabilities. 

Testability  reviews  have  been  conducted  by  the  Navy  at  NA  WCAD,  Lakehurst,  NJ 
using  the  SSM  on  over  2S00  weapon  subsystems.  The  SSM  analysis  indicated  that  CASS 
will  provide  approximately  90%  test  coverage  of  these  subsystems  based  on  the  input 
specifications  given  to  the  iiKxlel.  The  Navy  went  on  to  indicate  that  this  model  does  not 
have  an  ability  to  consider  revised  test  strategies  nor  the  addition  of  test  resources  in  TPSs. 
These  results  tend  to  substantiate  the  conclusion  of  the  test  equipment  technical  experts  at 
the  TTMs  that  modem  general  purpose  ATS  will  accommodate  95%  of  weapon  systems 
electronic  maintenance  technical  testing  requirements. 

8^  ATS  FAMILY  SYSTEM  ARCHITECTURE 

CASS  and  IFTE  use  flexible  industry  standard  architectures  that  promote 
modulariQr  and  partitioning.  CASS  and  IFTE  incorporate  open  architectures  with  MMS, 
VME,  and  Ethernet  interfaces.  These  architectures  provide  the  most  robust  interface 
capability  available  and  all  new  testing  requirements  that  have  been  identified  to  date  may 
be  accommodated  with  these  interfaces.  As  significant  performance  enhancing 
technologies  become  available,  the  families  can  incorporate  the  improvements  without 
disturbing  the  basic  architectures.  Upgrades  to  incorporate  new  technologies  should  be 
limited  to  those  technologies  required  to  meet  speciflc  DoD  test  needs.  For  example,  the 
insertion  of  a  VXI  chassis  or  the  additional  use  of  Ada  at  this  time  provides  no  specific  new 
test  capability. 

In  contrast  to  an  ATS  family  architecture,  the  Air  Force  is  using  a  series  of  evolving 
testers  that  grew  out  of  the  F-16  AIS  automatic  test  system.  While  the  F-16  ATE  (A/B,  C! 
D,  Advanced  Computer  AISs  and  the  lAIS),  the  B-1  ATE,  and  C-17  ATE  do  possess  high 
degrees  of  commonality,  they  do  not  exhibit  the  same  general  purpose  application 
flexibility  that  was  a  design  feature  of  IFTE  and  CASS.  If  they  had  a  more  general  purpose 
interface  with  robust  control  and  switching  capabilities  as  opposed  to  targeted  system 
specific  testing  interfaces,  they  might  well  have  been  viewed  as  an  early  ATS  family 
approach.  But  the  architectures  of  these  ATE,  as  designed,  make  them  dedicated  system 


^lecific  ATE;  and  as  such,  they  are  not  designed  for  general  purpose,  multi-system  testing 
applications  nor  suitable  for  a  wide  range  of  UUTs  from  the  system  level  down  to 
removable  cards  and  components. 

An  ATS  family  architecture  must  be  flexible  and  promote  modularity  and 
partitioning  of  the  principal  ATE  elements:  test  equipment  operating  system,  stimulus, 
measurement,  control  and  switch,  and  interface.  Equally  important,  the  architecture  must 
be  structured  such  that  the  interface  will  accommodate  multiple  general  purpose 
applications. 


9.  ATS  TECHNOLOGY  DEVELOPMENT  AND  EVOLUTION 


This  section  of  the  report  provides  an  analysis  of  ATS  technology  development  and 
technology  evolution  influences  that  have  a  direct  affect  on  ATS  used  by  the  Services.  The 
development  and  evolution  will  be  addressed  from  six  different  perspectives:  ATS 
industry,  standardization,  TPS  development  and  support  environment,  vertical  testability, 
Built-In-Testing  (BIT)  applications,  and  DoD  unique  ATS  needs. 

The  analyses  presented  in  this  section  touched  on  each  of  these  areas  and  produced  the 
following  general  findings.  Aside  from  the  consequences  of  anticipated  DoD  budget 
reductions,  ATS  industry  will  not  be  adversely  affected  by  a  common  ATS  family 
investment  strategy.  Standardization  efforts,  originating  from  both  the  military  and 
commercial  sectors,  have  provided  benefits  to  the  DoD  designated  ATS  families  by 
introducing  architectures  and  common  instruments  that  permit  design  and  use  flexibility. 
Advances  to  TPS  development  and  support  environments  will  reduce  future  TPS  costs,  and 
combined  with  standard  ATS  families,  wiU  improve  TPS  vertical  testability  by  increasing 
the  transportability  of  TPS  elements  across  iq>plications.  While  BIT  improvements  will 
enhance  system  diagnostics,  increasing  weapon  system  design  integration  and  conplexities 
will  preclude  substantial  reductions  of  the  total  DoD  need  for  off-equipment  automatic  test 
systems.  DoD  will  continue  to  possess  many  ATS  needs  that  are  unique  when  compared 
with  those  of  the  commercial  sector. 

9.1  ATS  INDUSTRY 

The  DoD  ATS  vendor  market  may  be  partitioned  into  three  aspects:  (1)  ATS  system 
integrators,  (2)  ATE  developers,  and  (3)  test  resource  (e.g.,  instmments,  power  supplies, 
digital  test  units).  The  test  resource  suppliors  are  mostly  second  or  third  tier  sub-contractors 
to  DoD  prime  equipment  or  prime  support  contractors. 

DoD  ATS  R&D  and  production  expenditures  are  evenly  spread  across  a  large  number 
of  contractors,  with  a  small  fraction  of  the  DoD  ATS  market  in  any  one  firm.  Table  67 
provides  an  example  of  the  range  and  depth  of  ATS,  ATE,  and  instrument  providers; 
however,  it  is  not  an  exhaustive  listing.  Like  with  any  set  of  products  and  manufacturers,  it 
is  difficult  to  assign  specific  ATS  industrial  capability  categories  that  fit  all  providers 
uniquely.  Therefore,  prominent  trends  are  indicated  based  on  the  observations  of  the 
authors. 


Table  67.  Early  1990s  ATS  Industrial  Capabilities 


ATE  SYSTEMS  INTEGRATORS 

COMMERCIAL 

ATE  SYSTEMS  & 

PRIMES 

ELECTRONIC  FIRMS/ 
ATS  HOUSES 

INSTRUMENTS/ 

MODULES 

BOEING 

AAI** 

FLUKE 

GM  HUGHES 

ALLIED  BENDIX** 

CHRYSLER  PENTASTAR** 

GENRAD 

GRUMMAN** 

*  HEWLETT-PACKARD 

EATON 

LOCKHEED 

EMERSON  (ESCO) 

*  RACAL-DANA 

MARTIN  MARIETTA 

GDE  SYSTEMS.  INC 

MCDONNELL 

GE** 

•  SCHLUMBERGER 

DOUGLAS 

HARRIS** 

TEKTRONIX 

NORTHROP 

HONEYWELL 

TELEDYNE 

RAYTHEON 

IBM 

ROCKWELL 

LORAL 

*TERADYNE 

INTERNATIONAL 

UNISYS 

WESTTNOHOUSE 

WAVETEK 

*  Mfgr  Special  Test  Eqmpmem  (STE)  Providers.  Note:  All  develop  own  in-house  STE. 
**  DoD  ATE  is  a  Major  Product  Line 


There  was  general  consensus  with  all  equipment  manufacturers  interviewed  that  the 
critical  test  equipment  technical  capability  resides  with  the  commercial  tester  and 
instrument  manufacturers  who  make  the  test  resources  which  are  integrated  into  ATS.  The 
commercial  tester,  instrument  manufactures,  and  test  resource  providers  develop  products 
to  meet  market  needs  for  commercial  customers.  Although  the  actual  ATS  market  shares 
are  proprietary,  and  therefore  not  presented  here,  the  vendors  observed  that  DoD  constitutes 
only  a  small  percentage  of  their  market.  For  example,  worldwide  test  instrument  sales  are 
over  $6  billion  annually  [VXI  1992].  It  is  estimated  diat  50%  of  the  manufactured  cost  for 
a  DoD  ATS  is  derived  from  COTS  test  resources.  The  cost  savings  economic  model 
presented  in  Section  5  of  this  paper  predicted  a  savings  that  averaged  out  to  $300  million 
annually  through  FY99.  Test  instruments  and  other  test  resources  represent  about  50%  of 
the  anticipated  savings  identified  by  this  macro  model.  These  savings  would  come  from 
quantity  reductions  and  economy  of  scale  factors.  The  net  result  is  a  potential  savings  of 
$50  million  per  year  which  is  less  than  0.5%  of  the  test  resource  COTS  market. 


On  the  other  hand,  most  electronics  firms  and  defense  primes  have  in-house 
organizations  which  develop  ATS  as  adjuncts  to  their  primary  products.  Only  a  few  major 
defense  corporate  divisions  and  other  firms  have  ATS  as  a  principal  business.  The  key 
technical  capability  common  to  these  businesses  is  complex  electromechanical  or  electro- 
optical  system  integration  and  packaging  to  DoD  requirements. 

DoD  test  program  software  houses  are  even  more  numerous  and  include  many  smaller 
businesses.  No  indications  were  found  that  emerging  ATS  technologies  will  reverse  this 
trend.  There  are  two  primary  ATS  software  categories  involved:  (1)  test  program  set  (TPS) 
development  support  software,  and  (2)  test  executive  software.  TPS  development  support 
software  is  provided  by  many  sources  and  typically  integrated  into  a  loose  knit 
development  environment.  Much  TPS  development  support  software  is  produced  by 
computer-aided  engineering  vendors  for  both  commercial  and  defense  markets. 

The  dominant  test  program  software  standard  (ATLAS)  was  jointly  developed  with  the 
Institute  of  Electronic  and  Electrical  Engineers  (IEEE)  in  cooperation  with  DoD,  and 
Defense  Ministries  of  NATO  countries,  airlines  and  U.S.  defense  contractors.  The  airlines 
through  Aeronautical  Radio  Incorporated,  ARINC,  developed  their  own  standard  which  is 
coordinated  with  the  IEEE  standard.  The  TPS  software  using  specific  applications  of  this 
standard  is  supplied  by  a  broad  set  of  defense  contractors  in  highly  competitive 
procurements. 

DoD’s  use  of  a  few  designated  ATS  families  should  improve  TPS  quality  by  providing 
a  smaller  set  of  target  ATE  systems  to  focus  TPS  development  process  and  third-party 
software  development  support  tools.  TPS  quality  has  the  potential  of  being  further 
enhanced  by  the  introduction  of  an  in^roved  standard  for  ATS  software  language  and 
improved  user-friendly  support  environments. 

The  authors  observed  that  ATS  expenditure  reductions  of  the  magnitude  outlined  above 
might  be  attributable  solely  to  DoD  budget  reductions.  An  ATS  family  investment  strategy 
was  found  to  provide  the  means  to  compensate  for  potential  budget  resource  reductions  and 
to  meet  weapon  system  ATS  needs,  both  without  adversely  affecting  operational  ATS 
capabilities  nor  adversely  affecting  ATS  industry. 

There  is  no  expectation  that  the  application  of  DoD  designated  ATS  families  or  the 
adoption  of  new,  improved  test  software  development  environments  will  adversely  affect 
industry.  Thwe  is  an  expectation  that  productivity  increases  should  be  expected  from  the 
TPS  industry. 


9J  STANDARDIZATION 

This  section  presents  a  short  discussion  on  the  evolution  of  ATS  standardization 
sq[)proaches  and  concludes  with  analysis  of  two  ATS  instrument  buses,  VXI  and  MMS. 

9.2.1  ATS  Standardization  Evolution 

A  number  of  standardization  efforts  and  approaches  have  been  applied  to  ATS.  The 
Services  have  initiated  various  standardization  approaches  to  reduce  duplication, 
obsolescence  upgrade,  and  operating  and  support  costs  generated  from  unique  ATE.  In  the 
1970s  the  Navy  introduced  die  Versatile  Avionics  Shop  Test  (VAST)  system  as  its  standard 
ATE.  In  the  early  1980s,  the  Air  Force  introduced  its  Modular  Automated  Test  Equipment 
(MATE),  sometimes  referred  to  as  a  system  of  standards.  In  the  1980s,  the  Army  began  the 
IFTE  program.  Now  in  the  1990s,  the  Navy  is  about  to  begin  operational  fielding  of  CASS. 

In  addition,  there  are  a  number  of  other  ATS-related  standardization  efforts  that  are  on¬ 
going  in  areas  below  the  ATS  system  level.  Many  of  these  efforts  have  yet  lower  levels  of 
standardization  opportunities.  Figure  27  illustrates  several  standardization  opportunities 
that  are  below  the  system  level.  These  include  major  functional  elements  (usually 
represented  by  hardware  components),  architectures,  test  languages.  Table  68  provides  a 
list  of  many  standards  currently  in  use,  and  indicates  if  the  standard  is  controlled  by  the 
military  or  the  commercial  seaor,  and  the  Service  using  tfte  standard.  This  list  represents 
many  of  today’s  major  ATS-related  standards;  however,  it  is  not  all  inclusive. 

The  newest  standard  ATS,  CASS  and  IFTE,  were  designed  to  take  advantage  of 
modularity  and  flexible  architectures.  Both  of  these  ATS  take  advantage  of  (TOTS  standard 
components.  Therefore,  CASS  and  IFTE  can  be  characterized  as  standard  ATS,  that  are  in 
themselves  broad  collections  of  proven  standards. 

Both  the  CASS  and  IFTE  program  office  personnel  indicated  that  the  flexibility  and 
upgradability  attributes  of  these  ATS  are  due,  in  part,  to  the  use  of  COTS  and  de  facto 
standard  components.  The  high  percentage  of  commercial  items  (approximately  65%  for 
both  CASS  and  IFTE)  permitted  the  design  teams  to  focus  on  ATS  system  requirements 
and  integration.  This  approach  left  the  design  and  production  of  very  accurate  and  precise 
stimulus  and  measurement  equipment  to  instrument  manufacturers  who  possessed  the 
needed  advanced  technical  expertise.  Additional  acquisition  side  benefits  include  improved 
quality,  higher  reliability,  no  development  costs,  and  lower  production  costs  (attributable 
to  competition  and  economies  of  scale). 
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Figure  27.  ATS  Standardization  Opportunities 


The  authors  observed  that  CASS  and  IFTE,  by  design,  addressed  weaknesses  found  in 
other  ATf  standardization  efforts.  They  use  flexible  industry  standard  architectures  that 
promote  modularity  and  partitioning  (e.g.,  MMS,  VME,  and  Ethernet).  As  performance 
enhancing  technologies  become  available,  these  families  of  testers  can  incorporate  the 
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Table  68.  Summary  of  Sdected  ATS  Related  Standards 
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improvements  without  dishirbing  basic  architectures.  The  authors  observed  that  upgrades 
to  incorporate  new  technologies  should  be  limited  to  only  DoD  test  needs.  For  example,  the 
insertion  of  VXI  or  Ada  at  this  time  provides  no  specific  new  test  or  cost  benefits  to  either 
the  CASS  or  IFTE.  Equally  important,  if  VXI  or  Ada  should  be  required  in  the  future, 
current  architectures  are  flexible  enough  to  accommodate  without  drastic  design  revisions. 

The  authors  concluded  that  the  use  of  a  DoD-defined  set  of  specifications  and  interfaces 
to  guide  the  evolution  and  transition  to  new  capabilities  will  help  in  economically 
accommodating  technological  change.  In  this  light,  DoD  should  not  only  establish  standard 
ATS  families,  DoD  should  define  key  family  elements  and  interface  standards  to  be  used 
when  implementing  ATS  families  to  help  manage  their  technology  evolution. 

Applying  a  standard  family,  along  with  the  related  subset  architectural  and  instrument 
standards,  is  fully  consistent  with  integrated  diagnostic  approaches.  For  example,  when  a 
contractor  determines  that  ATS  are  needed  as  part  of  the  most  effective  weapon  diagnostic 
mix,  the  proper  ATS  family  may  be  matched  to  this  need.  This  approach  supports 
concurrent  engineering  in  that  selecting  the  target  ATS  family  at  Milestone  II  allows  an 
earlier  focus  on  the  test  and  manufacturing  aspects  of  the  weapon  system  design.  The 
standards  also  reduce  the  risks  of  developing  test  software  programs  on  new,  design- 
unstable  testers. 

9,2,2  Instrument  Busses:  VXI  &  MMS 

VXI  and  MMS  are  two  test  instrument  bus  standards  used  by  the  DoD.  VXI  stands  for 
^MEbus  (a  computer  backplane  bus  standard)  E2Ltensions  for  ^strumentation.  VXI  was 
developed  by  a  consortium  of  companies  and  has  become  an  IEEE  standard.  Potential  VXI 
standard  changes  are  controlled  by  IEEE  under  its  balloting  processes.  MMS  stands  for 
Modular  Measurement  System,  and  was  developed  by  Hewlett  Packard.  Hewlett  Packard 
has  subsequently  turned  over  rights  and  controlling  ownership  of  this  standard  to  a 
consortium  of  companies.  Each  participating  company  is  entitled  to  only  one  vote 
regarding  potential  MMS  standard  revisions.  Although,  Hewlett  Packard  originally 
developed  MMS,  they  possess  only  one  vote  over  potential  standard  changes,  equal  to  other 
participating  consortium  companies. 

Both  of  the  standards  include  open  architectures  that  permit  the  interconnection  of 
modular  instruments.  The  specifications  for  these  standards  define  the  chassis,  mechanical 
and  electrical  interfaces,  cooling,  communications  protocols,  etc.  The  open  architecture  of 


both  standards  permits  compliant  products  from  different  manufacturers  to  interconnect 
and  operate  within  the  same  package  or  chassis. 

The  worldwide  market  shares  for  both  VXI  and  MMS  have  been  nearly  equal.  In  1991 
both  had  sales  of  approximately  $SS  to  $60  million  dollars.  An  early  1992  assessment 
projected  these  sales  to  increase  to  approximately  $80  to  $100  million  by  the  end  of  1992. 
More  recent  Prime  Data  information  indicates  that  the  VXI  1992  totals  may  be  higher,  or 
around  $146  million.  An  assessment  of  the  VXIbus  systems  market  in  1990  by  Prime  Data 
revealed  that  the  military  was  the  principal  user  (64%  on  a  systems  basis  and  44%  on  a 
module  basis).  The  same  1992  VXI  data  indicates  that  the  commercial  market  share  is 
growing  and  is  now  approximately  64%.  Assessments  of  MMS  markets  from  1989  through 
the  end  of  1992  from  Hewlett  Packard  estimated  that  54%  of  the  users  fall  in  the  Aerospace/ 
Defense  and  DoD/MoD  Programs.  [Source  data  are  compilations  of  information  provided 
by  VXI  Consortium  and  Hewlett-Packard,  both  referencing  data  provided  to  them  by  Prime 
Data.] 

The  standards  were  developed  primarily  to  meet  defense  needs  for  modular  ATE.  The 
operational  needs  that  lead  to  the  VXI  and  MMS  designs  are  found  mostly  within  DoD 
military  operational  environments.  Industry  representatives  (that  sell  VXI-  and  MMS- 
based  products)  indicated  that  in  contrast  with  DoD,  the  majority  of  the  corrunercial  sector 
users  do  not  have  specifre  requirements  for  downsized,  portable,  or  militarized  ATE. 
Product  selections  are  based  more  on  cost  and  performance  criteria.  Product  flexibility 
attributes  to  meet  specific  testing  applications  are  sometimes  used  as  criteria.  In  meetings 
with  instrument  manufacturers,  they  indicated  that  the  vast  majority  of  test  instrument  users 
in  the  commercial  sector  rarely  select  products  based  on  VXI  or  MMS  interfaces. 

Although  military  applications  have  (prior  to  1992)  accounted  for  the  greatest 
percentage  of  VXI  sales,  the  market  sales  for  other  commercial  electronics  sectors  are 
growiiig  faster.  While  portability,  ruggedaess  and  downsizing  are  not  the  driving  factor, 
other  benefits  of  performance,  open  standards,  flexibility,  and  time  to  market  are 
influencing  the  commercial  application  growth.  However,  many  VXI-based  products  are 
yet  to  be  developed,  such  as  a  comprehensive  set  of  RF  and  microwave  instrumentation. 

The  total  worldwide  test  instrument  market  sales  are  over  $6  billion  [VXI  1992] 
annually.  This  includes  many  product  categories  not  applicable  to  either  VXI  or  MMS  (i.e., 
oscilloscopes,  power  supplies,  X-Y  recorders,  etc.).  The  combined  sales  of  VXI  and  MMS 
for  1992  will  represent  less  than  2%  of  total  instrument  market  sales,  and  that  DoD  related 
sales  constitute  over  half  of  this  market  share.  However,  when  sales  are  compared  to  the 
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applicable  sub-segment  of  the  markets  each  potentially  support,  the  combined  VXI  and 
MMS  sales  percentage  increases  to  approximately  16%.  Even  at  projected  growth  rates, 
sales  to  DoD  of  these  two  jnxxluct  lines  will  constitute  the  largest  single  customer  for  the 
next  few  years. 

In  subsequent  meetings  with  Army,  Navy,  and  Air  Force  personnel,  there  was  general 
agreement  that  future  ATS  developments  or  modifications  should  not  limit  to  a  single  open 
architecture  standard,  but  rather  specify  system  test  performance  requirements  for 
identified  ^iplication  environments.  There  was  general  consensus  among  the  three 
Services  personnel  that  other  system-level  requirements  better  determines  bus  structures 
and  packaging  scheme  requirements.  Examples  include  requirements  for  an  open  systems 
architecture,  the  specific  stimulus  and  measurement  domain,  downsizing,  accuracy  and 
precision,  modularity,  interoperability,  and  upgradability.  The  following  summarize  the 
general  consensus  of  the  participants: 

a.  It  does  not  appear  the  VXI  or  MMS  standards  are  mutually  exclusive,  there  are 
valid  application  reasons  for  sevoal  architectures,  and  each  specification  is 
expected  to  have  future  growth. 

b.  VXI  and  MMS  architectures  were  originally  developed  in  response  to  DoD 
needs  and  had  migrated  to  the  commercial  sector. 

c.  DoD  should  specify  in  terms  of  requirements  as  opposed  to  solutions  (e.g.,  VXI 
or  MMS)  and  that  different  requirements  may  in  fact  drive  different  results. 

d.  DoD  should  not  exclude  either  VXI  or  MMS  solutions,  but  rather  DoD  should 
let  cost  and  performance  determine  applications. 

e.  Open  hardware  standards,  like  MMS  and  VXI,  provide  the  widest  commercial 
base  to  address  DoD  general  purpose  ATS  needs  and  offer  opportunities  to 
reduce  life  cycle  costs. 

The  following  sections  are  intended  to  sununarize  major  attributes  and  support  the 
conclusion  that  different  requirements  often  stimulate  different  engineering  trade-offs.  VXI 
and  MMS  have  technical  merit  depending  on  their  intended  use.  Open  architecture 
standards  like  VXI  or  MMS  are  highly  desirable  when  interoperability  of  modules  from 
multiple  venders  is  likely. 

9J22.1  VXI 

The  initial  VXI  standard  was  jointly  developed  by  five  companies  in  1987.  Since  that 
time  it  has  become  an  IEEE  Standard  (llSS-1992).  There  are  currently  over  60  VXI 


manufactuim  with  over  500  identified  prodiK^ts.  The  VXI  standard,  as  its  name  implies,  is 
an  extension  of  the  VME  microcomputo'  backplane  bus  standard  (^lecifies  mechanical  and 
functional  structure).  The  following  details  are  quoted  from  the  VXIbus  system 
specification: 

The  VMEbus  is  an  open  system  architecture  primary  focused  at  computer 
systems...  VMEbus  modules  are  approximately  six  inches  deep  and  come  in 
two  heights,  about  four  inches  and  nine  inches.  The  VXIbus  specification 
refers  to  these  as  the  A  and  B  sizes  respectively. . .  The  A  size  board  has  a 
single  96  pin  connector  known  a  PI,  while  the  B  size  may  include  a  PI  and 
P2  connector.  Each  of  these  DIN  connectors  consists  of  three  rows  of  32 
pins  on  0. 1  inch  centers. . . 

VXI  retains  PI  and  the  center  row  of  P2  exactly  as  defined  by  VMEbus... 
However,  VXI  has  made  substantial  additions  to  the  VMEbus  specification 
oriented  towards  instrumentation  that  can  best  be  described  as  an  electrome¬ 
chanical  super-set  and  a  logical  subset  .  .  VXI  has  added  two  Eurocard 
module  sizes  of  about  13  inch  depth  referred  to  as  the  C  and  D  sizes.  These 
modules  are  9  and  14  inches  high  respectively,  and  are  placed  on  1.2  inch 
centers.  The  C  Eurocard  is  the  same  height  as  the  VXIbus  B  size  board,  and 
may  sport  both  PI  and  P2.  The  D  size  module  is  a  triple  high  Eurocard  that 
may  include  a  P3  connector  in  addition  to  PI  and  P2...The  VXIbus  adds  a 
lOMHz  ECL  clock,  ECL  and  analog  supply  voltages,  ECL  and  TTL  trigger 
lines,  an  analog  summing  bus,  a  module  identification  line,  and  a  daisy 
chain  structure  known  as  the  local  bus...  The  VXIbus  P3  connector  adds 
many  of  the  same  resource  types  as  described  for  P2,  but  is  aimed  at  higher 
performance  instrumentation.  Included  on  P3  is  a  100  MHz  clock  and  sync 
signal,  additional  power  pins  of  the  same  supply  voltages,  more  ECL  trigger 
lines,  and  24  additional  lines  (4S  pins)  of  daisy  chain  local  bus. 

9.22.2  MMS 

The  open  architectural  concepts  of  the  MMS  standard  were  developed  in  the  early 
1980s.  They  were  intended  to  provide  an  environment  for  microwave  measurement  using 
modular  instruments.  The  following  details  are  quoted  from  the  MMS  bus  system 
specification. 

Modularity  in  large  Automatic  Test  Equipment  (ATE)  systems  was  recog¬ 
nized  as  an  important  attribute  in  the  1980’s  because  of  potential  size  reduc¬ 
tion,  serviceability  improvements,  and  the  ability  to  reconfigure.  The  United 
States  Air  Force,  Navy,  and  Army  reflected  this  in  their  requirements  for 
ATE  systems.  Modular  instrument  solutions  for  digital  and  low  frequency 
analog  applications  became  possible  using  card-based  architectures.  Modu¬ 
lar  test  equipment  solutions  for  microwave  tq>plications  were  being  built  us¬ 
ing  MMS  equipment..  MMS  addresses  the  needs  of  microwave  test  systems 


and  other  iq)plications  where  modularity  and  sensitive  measurement  capa¬ 
bility  are  needed 

The  Modular  Measurement  System  architecture  is  based  on  standard  sized 
modules,  in  increments  of  a  minimum  module  width,  with  external  support 
ftmctions  provided  by  the  mainframe.  The  mainframe  provides  modules 
with  power,  cooling,  physical  structure,  and  digital  communication  capabil¬ 
ity. 

Two  methods  of  digital  communication  are  provided  by  the  Modular  Mea¬ 
surement  System  architecture.  A  common  IEEE-488.1  interface  is  provided 
to  each  module  interface  connector,  providing  standardized  computer-to-in- 
strument  interface.  The  Modular  System  Intoface  Bus  (MSIB)  provides 
general  purpose  high  speed  digital  communication  between  logical  modules 
and  also  provides  a  standardized  computer  interface.  The  MSIB  system  also 
defines  an  extension  between  mainfrtunes,  allowing  module  placement  to  be 
dependent  only  upon  instrument  and  measurement  system  requirements. 

Instrument  Bus  ComparistMi  (VXl  and  MMS) 

VXI  and  MMS  specification  standards  cover  modular  instruments  with  partially 
overlapping  stimulus  and  measurement  domains.  Because  it  evolved  from  a  computer 
backplane  bus,  VXI  is  generally  well  suited  for  digital  communications  and  testing. 
Principal  VXI  applications  are  digital,  mixed-signal,  and  analog  up  to  1  GHZ.  MMS,  on  the 
other  hand,  was  specifically  targeted  towards  RF  and  microwave  instruments.  Several  VXI 
and  MMS  attributes  are  compared  in  Table  69. 

Instrument  manufacturers  observed  that  both  standards  approaches  can  be  made  to 
work  and  work  well  in  the  primary  domain  of  the  other.  However,  making  them  perform 
well  in  the  principal  domain  of  the  other  often  requires  unique  designs  and  super-sets  of  the 
basic  standards.  When  these  special  design  characteristics  are  needed,  the  products  are 
rarely  (X)TS. 

9.3  TPS  DEVELOPMENT  AND  SUPPORT  ENVIRONMENT 

9.3.1  Magnitude  of  TPS  Development,  Support,  and  Rehost  Costs 

TPS  development  costs  were  found  to  vary  from  between  5%  and  26%  of  the  total  ATS 
acquisition  for  field  (I-level)  maintenance  applications.  In  contrast,  for  depot  applications 
where  fewer  quantities  of  testers  are  purchased,  the  TPS  development  costs  were  found  to 
have  a  wider  and  generally  increasing  range  from  8%  to  50%  of  the  total  ATS  acquisition 
(previously  discussed  in  Section  4.3.2).  Although  specific  TPS  support  costs  could  not  be 
determined  from  the  collected  data,  evidence  collected  indicated  that  TPS  support  costs  are 


Tiible  69.  Comparison  of  Selected  VXl  and  MMS  Attributes 


ATTRIBUTES 

VXI 

MMS 

Size 

Based  on  C-Size: 

a)  Module  to  Rack  Height:) 

(\12iinTallUainframe)  1.02iii 
(12  +  ISIotOController)  Slot 

b)  Available  Volume  Per  SkH: 

146/V 

//9.187iiiXlVl.lUi<ix013.386iii  =  — — 

Slot 

(Multiple  sizes  available:  “B”  used 
-10%.  “C”  used  -80%.  “D”  used  - 10% 
of  time.) 

a)  Module  to  Rack  Height: 

(1  inTallmainframe)  0.875j»i 

{tUodultSlots)  Slot 

b)  Available  Volume  Per  Slot: 

3 

H4.457i«x  W1.827inxD18.386i/i  = 

Slot 

Cost 

Chassis  costs  are  comparable:  MMS  mainframe  costs  a  little  more;  however,  VXl 
requires  a  0>Slot  controUo'  (MMS  -  S9(X)/Slot  &  VXl  -  S800/Slot).  The  module 
costs  are  geneially  the  same  relative  cost  per  function. 

Maiket  Share 

-  8%  of  related  test  instrumentation 
maiket  DoD/aeroqtace  acquires  -1/2. 

-  8%  of  related  test  instrumentation 
market  DoD/aeroqrace  acquires  -1/2. 

Digital  Reference 

10  MHz  clock  (B-D  size  cards)  and  100 
MHz  clock  (D  size  card) 

10  MHz  clock 

EMC:  Radiated 
Module  to 
Module 

MMS  and  VXI  af^lications  require  near  field  magnetic  radiated  and  susceptibility 
compliance  and  specified  limits  similar.  VXl  has  reduced  susceptibility  area  and 
may  require  adapters  ot  module  to  module  placement  constraints  depending  on 
module  size  and  plication. 

EMC;  Radiated 
Mainfinameto 
Module 

Does  not  have  mainframe  requirements. 

&  has  reduced  area  available  for 
suscqttible  circuits. 

Mainframes  comply  with  near-field 
specifications 

Cooling 

Air  flow  specified;  however,  integrator 
of  modules  may  need  to  amsido’ 
differences  between  mainframes  and 
modules.  (Rev  1.4  defines  cooling  on  a 
slot/module  basis.) 

Air  flow  path  and  minimum  air  flow  per 
module  specified  (high  degree  of 
rqieaiability  between  modules  and 
mainframes) 

EMI 

Implementation  of  most  RF  noise 
constraints  beyond  VXI  standard  &  left 
to  module  integrator  to  implement  or 
incorporate  necessary  physical  features. 

Physical  specifications  and  required 
electrical  shielding  developed  to 
minimize  in  the  RF  environment 

>^bration  Noise 
(Microphonic) 

Not  specified 

0.010  g  limit 

Status 

IEEE  Standard.  Over  240  mfgr’s  IDs 
assigned.  Approximately  5-years  old.  Is 
not  optimiz^  for  RF;  therefore,  number 
of  RF  products  very  limited. 

Standard  controlled  by  consortium  of  9 
companies,  consortium  considering 
submitting  to  IEEE  for  standard 
consida:^on.  Approximately  8-years 
old.  Large  product  base. 
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also  substantial.  One  final  TPS  cost  category,  also  difficult  to  quantify,  includes  the  costs 
of  update  and  rehost  to  new  or  substantially  modiHed  ATS.  Information  provided  at  the 
TIMs  suggests  that  these  updating  and  rehosdng  costs  are  of  the  same  relative  magnitude 
as  the  original  TPS  development  Applying  estimates  of  these  TPS  development  support, 
and  rehost  cost  distributions  to  total  projected  ATS  acquisitions  (estimated  earlier  in 
Section  6)  indicates  that  these  costs  will  range  between  $3  and  $4  billion  over  a  6-year 
budget  cycle. 

93  J.  Opportunities  to  Reduce  TPS  Costs 

The  option  cost  factors  (developed  in  Section  5. 1.1. 2)  hypothesized  near-  and  long¬ 
term  improvements  to  current  approaches  used  to  develop  and  support  TPSs.  The  near  term 
improvements  were  estimated  to  reduce  the  development  type  costs  by  approximately  17% 
and  they  increase  to  70%  over  the  long  term. 

Although  these  cost  avoidance  benetits  do  not  exist  at  this  time,  there  is  evidence  that 
improvements  of  this  magnitude  are  feasible.  Results  from  an  Air  Force  laboratory  study 
titled  “Tester  Independent  Support  Software  System”  (TISSS)  provided  an  example  of 
potential  test  development  cost  reductions.  TISSS  qrplies  an  automated  approach  based  on 
new  digital  design  descriptions  and  tools  coded  in  the  VHSIC  Hardware  Description 
Language  (VHDL).  The  TISSS  approach  permitted  more  efficient  design  capture  and 
analysis,  facilitated  the  reuse  of  test  specifications  and  test  coding  details,  and  accelerated 
the  test  program  integration  and  acceptance  phases  of  the  development  cycle.  The  TISSS 
approach  specifically  touches  on  seven  of  the  eight  TPS  development  cycle  elements  (aU 
but  the  interface  device  design  as  previously  summarized  in  Table  27).  The  TPS 
development  process  elements  influenced  by  TISSS  compose  87%  of  the  identified  total 
process. 

The  data  from  the  following  two  charts  come  from  TISSS  program  analyses.  Figure  28 
illustrates  representative  test  development  complexity  growth  arising  out  of  increasingly 
more  complex  digital  designs.  The  arrow  in  this  figure  indicates  the  increasing  test  vector 
count  resulting  from  increasing  device  complexity.  Figure  29  iUustrates  the  potential 
development  time  reductions  that  may  be  achieved  if  new  test  program  development  and 
support  environments  become  a  reality.  Metrics  of  microelectronic  device  complexity 
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illustrated  by  this  figure  include  the  gate  count  of  the  device  and  the  number  of  input/output 


Figure  28.  Growing  Systems  Complexity  and  Test  Vector  Growth  # 


Figure  29  presents  the  results  of  comparing  the  test  program  development  times  for  the 
conventional  development  process  and  for  the  TISSS  method.  The  man-years  required  for 
the  first  four  data  points  are  estimates  based  on  the  TISSS  method,  and  the  final  data  point 
represents  actual  results  fi'om  the  TISSS  program. 
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9  J  J  TPS  Development  Environment  Concept 

Ongoing  efforts  by  (1)  the  Air  Force  to  reduce  the  prolifontion  of  computer 
programming  languages,  (2)  the  Joint  DoD/Service  DALSCOM  (DoD  ATLAS 
Standardization  Committee),  and  (3)  the  IEEE  have  all  culminated  in  draft  specifications 
for  A  Broad  Based  Environment  for  Test  (ABBET).  ABBET  is  intended  to  be  an 
environment  for  developing,  maintaining,  executing,  and  improving  test  programs  used  on 
automatic  test  systems.  An  objective  of  die  ABBET  standardization  effon  is  to  improve  test 
software  quality,  and  reduce  test  software  initial  acquisition  and  subsequent  rehost  costs. 
Much  of  the  ABBET  benefits  will  be  achieved  through  standardization  that  will  support 
reuse  of  test  program  data  and  common  test  software  across  the  product  (UUT)  life  cycle 
from  design,  factory  testing,  and  operational  maintenance  testing. 

The  ABBET  concept  is  illustrated  in  Figure  30  which  depicts  a  layered  drawing  of 
engineering  process  domains.  Key  relationships  between  domains  are  indicated  at  the 
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bottom  of  each  layer.  These  layers  represent  engineering  processes  which  (I)  stan  with  a 
description  and  information  source  of  a  product  to  be  tested,  and  (2)  resolve  the  detailed 
behavior  of  instruments  and  test  resources  that  will  be  used  to  test  and  diagnose  the  product. 

The  ABBET  environment  is  not  one  specific  implementation  of  the  functions  described 
in  this  figure.  Instead,  one  of  the  principle  goals  of  ABBET  is  establishing  standards  for  the 
Services,  data  formats,  information  exchanges,  and  interfaces  that  constitute  an  ABBET 
environment  for  ATS.  With  those  standards  in  place  for  the  elements  of  ABBET,  any 
number  of  separate,  competitive,  yet  compatible  implementations  will  be  possible.  The 
standards  which  define  the  pieces  of  an  ATS  ABBET  environment  will  be  concerned  with 
identifying  what  is  standardized  already  instead  of  how  to  implement  standards. 

Major  elements  of  an  ABBET  implementations  will  focus  on  the  development  of  test 
program  information  that  is  independent  of  any  one  particular  suite  of  ATS.  Using  an 
ABBET  compliant  environment  to  develop  tester  independent  TPS  information  is  the  key 
to  reducing  the  cost  of  rehosting.  The  top-down  model  presented  in  Figure  30  shows 
existing  and  potential  test-related  standards  on  the  left-hand  side,  and  possible  practices  and 
libraries  using  these  standards  on  the  right-hand  side.  The  majority  of  the  standards 
activities  have  taken  place  in  the  IEEE.  However,  other  standards  bodies,  such  as  the 
Electronic  Industry  Association  (EIA),  have  worked  many  areas  of  computer-aided  design 
and  manufacturing.  A  third  standards  body,  the  International  Standards  Organization 
(ISO),  is  developing  Product  Description  Exchange  standards  for  mechanical,  electronics, 
shipsbuilding,  etc. 

However,  ABBET  is  at  an  initial  conception  and  formulation  stage.  More  R&D  across 
multiple  standards  areas  is  needed  in  order  to  gain  the  benefits  advertised.  However, 
benefits  to  both  industry  and  DoD  are  feasible,  and  interested  parties  are  seriously  pursuing 
this  development  A  draft  specification  has  been  balloted  by  the  IEEE  members  and  will  be 
undergoing  subsequent  standardization  development  and  review  this  next  year.  The  authors 
working  on  this  report  recognized  a  need  for  continuing  development  of  an  improved  test 
environment  and  identified  R&D  “plus-up”  resources  in  Section  6  towards  this  goal. 
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Figure  30.  A  Broad  Based  Environment  for  Test  (ABBET) 


9.4  VERTICAL  TESTABILITY 

Three  very  different  concepts  or  views  of  vertical  testability  are  present  within  the  ATS 
technical  communities;  measurement  accuracy  cones  of  tolerance,  TPS  transportability 
across  application  levels,  and  testing  correlation  across  application  levels.  The  first  is  well 
understood  and  involves  a  cone  of  calibration  accuracy  tolerances  (Tolerance  Accuracy 
Ratios  or  TARs)  between  test  equipment  and  calibration  standards.  With  the  proper  control 
of  the  TARs,  the  absolute  accuracy  requirements  between  measurements  (such  as  various 
maintenance  levels)  may  be  controlled.  The  second  area  is  the  least  well  defined  or 
understood,  and  involves  the  ability  to  move  TPSs  both  among  and  across  functional  testing 
levels:  from  factory  to  lower  maintenance  levels  [VERTICAL  1992].  The  third  area  is  the 
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relative  testing  and  diagnostic  accuracy  between  applications,  usually  maintenance  levels 
[MC  COMB  1992]. 

Because  of  the  impact  to  an  ATS  investment  strategy,  this  analysis  focuses  primarily 
on  the  TPS  transportability  aspects  of  vertical  testability.  Finally,  this  section  concludes 
with  a  discussion  of  the  third  area,  the  relative  diagnostic  accuracy  improvements  that  may 
be  associated  with  the  adoption  of  common  ATS  families.  The  first  view  involving 
calibration  accuracy  and  TARs  involves  basic  measurement  sciences.  This  area  was  not 
separately  analyzed  as  part  of  this  study;  however,  aspects  of  calibration  accuracy  are 
discussed  within  the  Relative  Diagnostic  Accuracy  Test  Result  Correlation  Section  7.4.2. 

9.4.1  TPS  Transportability 

Vertical  testability,  as  principally  addressed  in  this  study,  represents  a  notional  measure 
of  how  well  some  or  all  of  a  product’s  test  program  set  (TPS)  elements  may  be  reused  at 
different  testing  levels.  The  primary  TPS  elements  are  the  test  program  software,  testing 
documentation  and  instructions,  and  interface/test  adapter  hardware.  The  reusability  of 
these  elements  is  directly  tied  to  three  critical  ATS  design  factors;  Test  Strategy,  ATS 
Flexibility,  and  Data  Transportability. 

a.  Test  Strat^es  directly  influence  design  characteristics  of  the  TPS  elements. 
These  strategies  are  based  on  the  intended  purpose  of  desired  tests,  and  very  fre¬ 
quently  the  specific  testing  purpose  is  linked  with  the  application  level  (e.g.,  fac¬ 
tory,  depot,  or  operational).  In  general,  it  is  easier  to  transport  TPS  elements 
from  one  ATS  to  another  if  similar  test  strategies  are  employed. 

Test  strategies  generally  fall  into  three  categories:  Process  Verification,  Product 
Verification,  and  Diagnostic  (or  Fault  Detection/Isolation).  Depending  on  the 
intended  application  level,  one  or  more  of  the  strategies  may  be  used.  For 
complex  systems,  the  number  of  tests  may  range  upwards  to  the  hundreds  or 
even  thousands.  Also,  the  order  in  which  the  individual  tests  are  conducted  may 
vary,  depending  on  the  test  strategy  used.  Due  to  the  unique  objectives  of 
different  strategies,  specific  design  characteristics  of  individual  ATS  elements 
may  have  substantive  differences  that  impede  vertical  testability. 

The  following  summarizes  principal  differences  between  test  strategies.  Process 
Verification  test  strategies  are  intended  to  keep  a  manufacturing  or  assembly 
process  in  statistical  control.  Product  Verification  Test  Strategies  are  intended  to 
indicate  the  status  of,  or  verify  that,  a  product  (or  subcomponent  during  a 
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manufacturing  process)  meets  specified  characteristics.  Diagnostic  test 
strategies  are  intended  to  detect  and  isolate  faults,  and  link  identified  faults  to 
corrective  actions.  In  addition,  each  of  these  basic  strategies  may  be  further 
implemented  with  a  parametric  or  functional  testing  approach.  The  parametric 
methodology  involves  the  sequential  application  of  prescribed  stimulus  and 
quantitative  measurements  to  prescribed  limits,  whereas  the  functional 
methodology  involves  verifying  an  ability  to  perform  intended  functions  when 
subjected  to  emulated  operational  signals. 

b.  ATS  Flexibility  describes  an  ability  of  the  testing  system  to  adapt  to  changing 
test  requirements  or  to  incorporate  additional  capabiliti'^s.  In  general,  it  is  easier 
to  transport  TPS  elements  firom  one  ATS  to  another  of  equal  or  greater  flexibility 
provided  they  both  have,  or  may  incorporate,  capabilities  to  cover  the  same 
testing  requirements  spectrum. 


The  effect  of  ATS  flexibility  on  the  ease  of  transporting  TPS  elements  may  be 
addressed  from  several  flexibility  perspectives:  the  ability  of  the  ATS  to 


accommodate  new  requirements  at  flexible  physical  interfaces,  the  ability  to 
accommodate  the  same  range  of  stimulus  and  measurement  performance 
properties  across  ATE  versions,  and  the  degree  to  which  a  TPS  specification 
language  will  accommodate  common  syntax  and  common  testing  methods. 
Vertical  transportability  of  TPS  elements  is  severely  impeded  when  the  target 
ATS  are  neither  identical  (or  nearly  so)  nor  incorporate  features  that  foster  ATS 
flexibility. 

Data  IVansportability  is  an  ability  to  obtain  and  reuse  information  in  a  digital 
format  that  describes  the  design  of  the  product  being  tested,  the  capabilities  and 
characteristics  of  the  source  and  target  ATS,  and  the  testing  strategies.  In 
general,  it  is  easier  to  transport  TPS  elements  from  one  ATS  to  another  if  there 
is  an  automated  mapping  of  data  fields,  ATS  capabilities  and  characteristics, 
physical  connections,  and  alternative  testing  strategies. 

Full  one-to-one  compatibility  of  TPS  elements  from  one  tester  (or  application 
level)  to  another  is  rare,  and  even  in  the  best  of  cases,  some  mapping  of  data 
fields,  physical  connections,  or  alternate  strategies  are  required.  The  ease  of  this 
mapping  is  dependent  on  how  well  this  information  has  been  captured, 
recorded,  and  verified.  In  some  cases  this  information  mapping  may  even  lend 
itself  to  automation.  If  this  data  is  in  neither  a  retrievable  or  transferable  form. 
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some  degree  of  reverse  engineering  will  be  required  to  rehost  TPS  elements  on 
different  ATS. 

In  general,  vertical  testability  will  improve  when  the  impediments  directly  related  to  the 
critical  ATS  design  factors  are  reduced.  The  authors  concluded  that  the  following  three 
essential  actions  are  needed  to  reduce  impediments  and  improve  vertical  testability. 

(1)  Test  strategies  should  take  into  account  the  intended  purpose  of  needed  test¬ 
ing  at  all  levels  where  the  TPS  elements  may  be  used. 

(2)  ATS  flexibility  may  be  addressed  from  two  perspectives:  standardized  ATS 
(i.e.,  a  common  ATS  family)  or  flexible  architecture,  general  purpose  ATS. 
The  standard  ATS  approach  avoids  the  need  to  account  for  differences, 
because  the  equipment  is  identical  at  all  levels.  Since  it  is  unlikely  that  full 
standardization  could  (or  should)  be  achieved,  highly  flexible  architectures 
that  accommodate  a  wide  range  of  general  purpose  testing  needs  are  most 
desirable.  This  is  presently  achievable  with  the  CASS  and  IFTE  general  pur¬ 
pose  ATS  families  of  testers. 

(3)  Given  that  full  one-to-one  compatibility  of  either  ATS  hardware  or  test  strat¬ 
egies  will  not  be  achievable  for  a  wide  range  of  applications,  the  greatest 
overall  improvements  to  vertical  testability  will  come  from  the  creation  of  a 
software  development  and  support  environment  that  minimizes  reverse 
engineering  tasks  for  rehosting  TPS  elements  across  ATS  and  testing  levels. 

9A2  Relative  Diagnostic  Accuracy  Test  Result  Correlation 

The  relative  diagnostic  accuracy  between  maintenance  levels  (i.e.,  organizational, 
intermediate,  and  depot)  is  measured  in  terms  of  RTOK,  CND,  and  NEOF.  These  stand  for 
ReTest  OK,  Can  Not  Duplicate,  and  No  Evidence  of  Failure  respectively.  They  are 
generally  applied  when  an  item  is  checked  out  at  the  next  level  of  maintenance  in  a  repair 
chain  and  the  alleged  problem  can  not  be  found  nor  duplicated.  Adverse  consequences  of 
diagnostic  inaccuracies  include  increased  maintenance  work  load,  increased  spares 
logistics  pipeline,  increased  support  costs,  and  reduced  systems  availability.  Although  not 
the  only  source  of  these  problems,  ATS  must  be  considered  one  of  the  primary  contributors. 
In  this  context,  vertical  testability  is  a  metric  of  how  well  testing  results  correlate  at 
different  maintenance  levels. 

This  form  of  vertical  testability  is  directly  influenced  by  five  testing  process  conditions 
that  may  vary  across  maintenance  levels:  test  strategy,  test  equipment,  weapon  subsystem 


configuration  level,  pass/fail  thresholds,  and  calibration  approaches.  (There  are  yet  other 
non-ATS  driven  testing  process  condition  differences  that  can  affect  diagnostic  accuracy, 
and  these  will  be  discussed  Iniefly  in  a  subsequent  paragraph.)  These  testing  process 
differences  can  result  in  nteasured  variables  that  indicate  a  good  item  at  one  maintenance 
level  and  yet  indicate  a  failed  item  at  another  maintenance  level  (or  visa  versa).  In  order  to 
assess  how  this  might  occur,  a  simple  model  of  a  subsystem  soft  failure  mode  is  used  as  an 
illustrative  example. 

Many  failure  modes  do  not  fall  into  the  category  of  a  “hard  failure”  (i.e.,  a  bumt-out 
resister  or  an  open  diode).  Instead,  an  item  may  exhibit  a  “soft  failure,”  where  there  are  only 
slight  changes  in  critical  output  performance  levels  such  as  time,  current,  or  voltage.  In 
most  cases,  a  gradual  degradation  of  performance  levels  within  some  tolerance  is 
acceptable,  and  will  not  significantly  affect  desired  performance  of  the  full-up  system. 
However,  once  the  performance  of  an  item  varies  beyond  certain  limits  and  some  aspect  of 
the  full  up  system  performance  falls  below  a  required  capability,  both  the  system  and  the 
item  are  said  to  be  experiencing  a  soft  failure.  In  a  soft  failure  mode,  some  aspect  of  the 
system  still  may  be  partially  capable  of  meeting  mission  requirements. 

As  noted,  the  five  ATS  related  testing  process  conditions  may  influence  the  relative 
diagnostic  accuracy  (or  vertical  testability)  across  maintenance  levels.  The  following 
examples  illustrate  some  effects  of  these  differences: 

a.  A  system  may  be  tested  as  a  complete  entity,  or  assemblies  that  make  up  the  sys¬ 
tem  may  be  tested  as  separate  units.  The  configuration  level,  in  this  context, 
addresses  the  complete  hierarchy  of  elements  that  compose  the  system  (e.g., 
from  system,  to  a  subsystem,  to  a  component).  When  tests  are  conducted  at  dif¬ 
ferent  configuration  levels  (e.g.,  LRU,  SRU,  or  component)  at  different  mainte¬ 
nance  sites,  measured  parameters  are  often  influenced  by  other  circuits  present 
in  an  electrical  path.  The  electrical  circuits  undergoing  test  and  measurement  are 
determined  by  the  configuration  level  of  the  unit  being  tested.  In  some 
instances,  the  desired  parameter  may  not  even  be  accessible  while  combined 
with  its  next  level  of  system  assembly;  therefore,  a  totally  different  testing 
approach  must  be  used  at  each  location. 

b.  The  test  strategies  for  assessing  performance  at  various  maintenance  levels  are 
frequently  different,  resulting  in  different  reported  values  for  the  same  unit 
under  test.  For  example,  one  strategy  may  determine  a  level  of  performance  by 
measuring  power  around  a  specified  tolerance  range,  while  another  test  strategy 
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measures  current  over  a  related,  yet  different,  tolerance  range.  To  eliminate 
diagnostic  accuracy  error  sources,  these  differences  must  be  correlated  to  a  high 
level  of  precision.  Given  there  may  be  hundreds  or  even  thousands  of 
parameters  to  be  measured,  reference  correlation  between  measurements  is  not 
always  given  the  rigorous  attentions  required  to  prevent  error  sources. 

c.  Different  ATS  at  different  maintenance  levels  may  have  the  same  consequences 
as  item  b.  above. 


d.  Even  if  all  of  the  above  conditions  are  minimized  by  using  identical  ATS  and 
identical  test  strategies,  the  applied  tolerance  around  the  pass/fail  threshold  of  a 
specified  parameter  can  influence  diagnostic  accuracy.  The  pass/fail 
performance  threshold  may  vary  enough  at  different  maintenance  levels  to 
result  in  either  falsely  accepting  or  falsely  rejecting  an  item.  This  condition 
frequently  results  when  the  ability  to  measure  a  specified  parameter  to  a  level  of 
precision  is  less  than  the  accuracy  permitted  by  the  ATS,  the  test  strategy,  or  the 
calibration.  Due  to  the  sheer  number  of  potential  parameters  that  may  be 
measured  in  complex  electronics,  coupled  with  many  test  approaches  available 
to  a  test  engineer,  it  is  questionable  that  all  specified  test  tolerances  have  been 
rigorously  analyzed  to  ensure  overlapping  tolerances  at  different  testing  levels 
do  not  result 


The  testing  process  condition,  which  involves  overlapping  thresholds  of  pass/fail 
parameter  tolerances  at  different  maintenance  levels,  may  be  the  most  difficult  to  manage. 
To  manage  this  diagnostic  error  source  problem,  an  in-depth  knowledge  of  nearly  every 
parameter  measured  on  a  specific  unit  under  test  is  required.  Specific  details  must  include 
an  understanding  of  tolerance  guardbands  needed  for  each  parameter  and  at  each 
maintenance  level  where  the  parameter  might  be  measured.  In  general,  the  tolerances 
should  be  adjusted  such  that  the  least  accurate  (or  widest  tolerance)  exists  closest  to  the 
user.  The  goal  in  this  case  is  to  eliminate  die  possibility  of  a  specific  measured  parameter 
value  being  outside  the  pass/fail  tolerance  guardband  (or  a  failure)  at  the  maintenance  test 
level  closest  to  the  operator,  while  being  inside  the  pass/fail  tolerance  guardband  (or  a  pass) 
at  the  next  level  of  maintenance. 


Management  of  this  error  source  is  further  hampered  by  weapon  systems  that  require 
higher  levels  of  accuracy  and  precision.  As  needed  accuracy  and  precision  approach  current 
technology  limits  to  calibrate  test  instruments,  the  available  tolerance  guardband  at 
different  maintenance  levels  shrinks. 
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Based  on  discussions  with  both  designers  and  users  of  ATS  during  this  study,  the 
authors  were  left  with  the  impression  that  this  ^)ecific  problem  is  generally  overlooked  and 
only  addressed  when  identified  as  a  specific  problem  through  a  variety  of  maintenance  data 
collection  and  information  feedback  tools  found  in  the  Services.  Standard  ATS  families 
will  not  eliminate  this  error  source  problem;  however,  it  may  be  easier  to  identify  and 
isolate  these  problems  if  other  potential  error  sources  are  eliminated  from  contention  by 
using  the  same  equipment  at  multiple  levels. 

Finally,  the  arror  sources  from  the  other  four  testing  process  conditions  may  be  reduced 
or  even  eliminated  by  using  the  same  ATS  at  multiple  maintenance  levels.  Test  strategy, 
test  equipment,  calibration  approaches,  and  even  the  subsystem  connguration  level  may  be 
identical  (or  at  least  managed)  if  the  same  ATS  is  used  at  the  various  testing  sites. 
Furthermore,  other  testing  process  conditions  such  as  the  quality  and  thoroughness  of  ATS 
training,  consistency  of  calibration  procedures,  operator  errors,  communication  between 
maintenance  levels,  etc.,  may  be  better  managed  if  the  same  ATS  is  used  at  multiple 
maintenance  levels. 

Therefore,  the  authors  concluded  that  conunon  ATS  families,  when  used  at  multiple 
maintenance  levels,  will  tend  to  improve  the  relative  diagnostic  accuracy  across  different 
maintenance  levels. 

BUILT-IN-TEST  (BIT)  APPLICATIONS 

BIT  will  not  reduce  ATS  requirements  for  off-equipment  repair  over  the  next  decade. 
BIT  does  not  have  the  same  accuracy  and  range  of  testing  capabilities  as  modem  general 
purpose  ATS.  BIT  technology  development  is  not  keeping  pace  with  built-in-testing  needs 
of  new  systems  employing  emerging  technologies  due  to  missing  integration  of  necessary 
control  and  observation  pathways  and  sensors. 

BIT  is  a  key  element  of  current  and  evolving  on-equipment  maintenance  strategies  for 
fault  detection  and  fault  isolation  of  advanced  ground  electronic  and  avionic  systems.  Some 
future  maintenance  concepts  are  being  developed  on  the  premise  that  organizational  and 
intermediate  level  test  equipment  will  be  eliminated  by  improvements  in  reliability  and 
BIT.  Also,  BIT  may  be  used  to  support  factory  acceptance  and  depot  level  ATS-based  test 
But  designing  a  BIT  system  for  so  many  test  roles  is  quite  difficult  The  absence  of  BIT 
standards  in  each  of  these  test  roles  requires  custom  hardware  and  software  engineering 
which  is  difficult  to  achieve  across  a  multiple  contractor  base. 


This  section  provides  a  short  discussion  on  the  types  of  BIT,  and  provides  an  analysis 
of  why  current  DoD  systems  are  not  achieving  the  full  potential  of  BIT  capabilities  in 
present  designs.  This  is  followed  by  discussions  of  current  BIT  technology  limitations,  and 
how  BIT  and  ATS  might  collectively  be  used  to  improve  integrated  diagnostic  capabilities. 

9.5.1  T^pes  of  BIT 

There  are  two  basic  types  of  weapon  system  BIT.  Logic  level  based  BIT,  also  identified 
as  digital  BIT,  takes  advantage  of  digital  pathways  to  concurrently  or  off-line  verify  system 
performance  and  fault  isolate  to  a  removable  item.  The  second  type  of  BIT  is  sensor  and 
microcomputer  based  and  can  be  applied  across  all  electronic,  electro-optical,  and  electro¬ 
mechanical  applications. 

The  first  type  of  BIT  may  support  certain  ATS  applications.  The  second  type  of  BIT  is 
typically  not  effective  in  supporting  ATS  applications.  BIT  integrates  with  modest  effort 
into  subsystems  that  are  computer  controlled  and  makes  use  of  many  sensors  integrated  for 
system  control.  Systems  may  provide  stimulus,  and  most  definitely  provide  observation 
paths  necessary  for  effective  BIT.  But  when  examining  subsystem  components 
encompassed  by  BIT  at  a  system  level,  the  BIT  functions  and/or  information  often  are 
incomplete  and  disjointed.  The  BIT  requires  computation  and  sensors  in  place  acting  in 
unison.  In  general,  line  removable  units  and  lower  levels  of  indenture  components  cannot 
utilize  this  kind  of  on-equipment  BIT  to  supplement  or  replace  ATS  applications  for 
removable  item  testing. 

Both  types  of  BIT  will  increase  off-equipment  testing  needs  because  of  the  added 
circuitry  and  untestable  elements  within  the  BIT  design. 

9.5.2  BIT  Applications 

BIT  is  not  being  integrated  to  the  fullest  extent  possible  in  most  current  DoD  systems. 
The  reasons  are  several:  (1)  it  is  not  an  effective  add-on  feature  for  existing  systems,  (2) 
many  commercial  devices  do  not  support  BIT  capabilities,  (3)  BIT  does  not  come  free,  and 
(4)  there  are  some  technology  areas  where  BIT  has  not  matured  enough  to  meet  current 
needs.  This  section  will  discuss  the  first  three  reasons,  and  the  last  will  be  covered  in  the 
next  section. 

BIT  is  not  an  effective  add-on  to  existing  complex  electronic  systems.  Granted,  some 
minor  BIT  features  might  be  added;  however,  fuD  robust  fault  detection  capabilities  need 
to  be  designed  and  integrated  into  the  system  circuitry.  If  a  system  is  undergoing  major 


modifications,  the  portion  of  the  system  being  redesigned  may  incorporate  BIT.  However, 
the  BIT  that  is  being  added  must  be  consciously  designed  into  the  system.  Also,  it  is  not 
practical  to  use  BIT  to  test  each  parameter  or  condition  subject  to  failure,  since  each  BIT 
feature  has  its  own  overhead  burden  (space,  heat,  interconnections,  software/fumware, 
etc.).  Implementation  of  an  on-board  testing  capability  that  provides  full  fault  detection 
over  the  system  operating  range  would  require  the  incorporation  of  some  major  subset  of  a 
full  sized  general  purpose  ATS  on  each  circuit  card.  This  also  is  valid  at  all  design  levels 
down  to  and  including  individual  components.  BIT  only  checks  for  a  unique  problem  or  set 
of  problems  that  might  be  detectable  within  a  subsystem.  There  is  a  wide  range  of  problems 
within  each  BIT  implementation  that  are  non-detectable. 

Some  of  the  electronic  components  planned  for  the  advanced  avionics  of  the  F-22 
aircraft  were  reviewed.  Many  of  these  proposed  digital  microelectronic  devices  are 
commercial  or  militarized  versions  of  commercial  products,  and  they  do  not  include 
architectures  and  features  compatible  with  BIT  (such  as  boundary  scan  and  diagnostic  bus 
interfaces).  Consequently  major  portions  of  critical  line  replaceable  module  (LRM) 
circuitry  (often  controlling  portions)  either  must  be  void  of  BIT  or  must  be  redesigned  to 
incorporate  BIT  capabilities.  Due  to  cost  and  programmatic  risk  concerns,  major 
developments  other  than  the  F-22  which  also  involve  complex  electronics  will  use  high 
relative  percentages  of  commercial-based  microelectronic  devices.  Therefore,  until  the 
commercial  sector  demands  more  BIT  features,  full  digital  BIT  will  rarely  be  as  extensive 
as  proposed. 

There  are  both  direct  and  indirect  costs  associated  with  BIT.  BIT  adds  complexity  to 
the  circuits.  It  introduces  additional  testing  requirements.  Total  device  reliability  is  reduced 
since  there  are  more  microcircuit  elements.  BIT  is  also  another  source  of  diagnostic 
accuracy  errors.  Once  implemented  in  a  design,  BIT  diagnostic  accuracy  problems  are 
difficult  and  often  costly  to  eliminate  since  they  require  system  configuration  changes. 
Finally,  redesigns  of  commercial  devices  to  incorporate  BIT  must  be  funded;  and  once 
incorporated,  the  benefits  of  adopting  a  commercial  based  device  are  reduced  (i.e., 
economies  of  scale,  quality,  yield).  Additional  problems  with  BIT  in  other  domains  are 
discussed  in  the  next  section. 

9,53  BIT  Technology  Limitations 

The  performance  requirements  for  BIT  are  often  set  at  very  high  levels  in  terms  of 
technical  parameters  like  fault  coverage.  Perhaps,  even  a  more  user-oriented  criterion  like 
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mean-timc-to-repair  may  be  applied  to  measure  the  performance  of  BIT  based  designs.  Yet 
BIT  suffers  from  the  designers’  inability  to  truly  assess  the  effectiveness  of  BIT  designs 
until  after  fielding  a  system.  Other  BIT  technology  limitations  include  the  following; 

a.  The  criteria  for  successful  BIT  design  is  difficult  to  relate  to  measurable  user- 
oriented  performance  criteria  in  each  test  role  before  design  completion. 

b.  The  user  oriented  performance  criteria  are  not  related  to  the  physical  fault 
mechanisms.  Physical  fault  consideration  allows  the  designer  to  take  into 
account  the  probability  of  failure  in  assessing  overall  BIT  effectiveness.  BIT 
models  that  consider  fault  mechanism  probabilities  and  user-oriented 
performance  criteria  are  not  well  explored  except  for  very  critical  subsystems 
analyses  that  might  be  performed  as  part  of  a  Failure  Mode,  Effects,  and 
Criticality  Analysis  (FMECA). 

c.  BIT  is  most  effective  in  digital  electronic  subsystems  where  every  circuit  path 
can  serve  as  a  sensor  and  stimulus  for  built-in  test  But  digital  BIT  typically 
cannot  measure  large  classes  of  faults  that  are  common  to  electronic  circuits 
such  as  non-detectable,  timing,  and  some  open  circuit  faults.  Further,  even 
considering  what  are  called  stuck-at-logic-level  faults,  BIT  performance 
analysis  ignores  circuit  nodes  which  arc  untestable  by  design  and  reports 
metrics  against  those  faults  that  BIT  could  have  detect  This  leaves  a  large  gap 
between  a  narrowly  dehned  technical  metric  and  a  user-oriented  performance 
metric. 

d.  BIT  design  for  fault  isolation  requires  the  addition  of  electrical  and  physical 
isolation  elements  added  to  the  subsystem.  These  additions  may  decrease 
system  reliability.  BIT  designers  focus  on  on-equipment  test  requirements, 
recognizing  that  off-equipment  BIT  requirements  cannot  be  met  without 
affecting  weight,  reliability,  or  engineering  cost 

For  example,  in  1988  during  the  preliminary  design  of  the  F-22  Environmental  Control 
System,  it  was  found  that  the  addition  of  sensors  beyond  those  required  for  mission 
performance  actually  decreased  system  reliability.  Even  though  the  BIT  and  sensor 
redundancy  might  have  afforded  increased  availability,  they  were  deleted  from  the  design 
to  achieve  reliability  goals.  A  careful  analysis  concluded  that  BIT  based  on  mission  sensor 
and  microcomputer  resources  would  be  more  reliable,  yet  less  effective.  [KOTO  1989] 
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The  ability,  permitted  by  current  BIT  technology,  to  detect  faults  rapidly  degrades  as 
the  required  testing  spectrum  broadens  to  mixed  signal  or  analog.  For  example, 
requirements  for  high  RF,  microwave,  and  many  analog  domains  exceed  current  BIT 
technical  capabilities.  Yet  future  weapons  will  have  a  majority  of  subsystems  that  fall  in  the 
domains  where  BIT  is  either  limited  or  impossible  at  this  time.  The  following  are  projected 
percentages  for  the  F-22  and  A~12  aircraft  avionics  systems: 

vl)  F-22:  24%  digital,  43%  analog/mixed,  33%  RF/EO 
(2)  A- 12: 42%  digital,  33%  analog,  15%  RF/EO 

9.5.4  Integrated  Diagnostic  Applications 

Decisions  regarding  fault  detection,  fault  isolation,  and  repair  verification  should  not  be 
limited  to  BIT  and  ATS,  but  radter  should  embrace  a  systems  design  and  trade-off  decision 
process  that  addresses  the  advantages  and  disadvantages  of  BIT  and  ATS,  along  with 
projected  reliability,  mission  criticality,  and  safety  considerations  of  each  device,  unit  or 
subsystem.  For  example,  microelectronic  subsystem  reliability  growth  has  been  a  major 
factor  in  enhancing  BIT  attractiveness  for  operational  applications.  Due  to  the  complexity 
and  expansive  interconnection  between  subsystems,  BIT  may  provide  a  good  indicator  for 
a  technician  to  begin  trouble-shooting  a  problem.  But  as  indicated  earlier,  BIT  is  poor  at 
fault  isolation,  even  in  the  domains  where  the  technology  is  mature. 

The  ATS  family  concept  is  fully  compatible  with  BIT  and  where  appropriate  should 
continue  to  exercise  internal  BIT  of  die  UUT  under  off-equipment  testing.  In  fact  if 
designed  appropriately,  BIT  may  reduce  repair  times  by  helping  assist  the  diagnostic 
process.  For  example,  BIT  may  be  used  to  steer  the  TPS  during  execution  on  an  ATE.  The 
diagnostic  process  is  essentially  a  divide-and-conquer  approach  that  isolates  and  narrows 
down,  through  the  design  hierarchy,  to  a  specific  replaceable  item.  Any  tool  that  supports 
the  process  and  can  do  dus  job  economically  (in  terms  of  acquisition  as  well  as  support 
costs)  should  be  considered.  For  example,  as  systems  become  more  complex,  and  as  the 
percentage  of  input/output  (I/O)  access  points  available  for  external  testing  relative  to  the 
number  of  items  on  a  UUT  are  reduced,  new  dedicated  diagnostic  I/O  features  will  be 
needed.  BIT,  in  combination  with  diagnostic  buses  and  on  device  matrix  switches,  may 
further  enhance  system  diagnostic  performance. 

Designers  should  not  think  in  terms  of  BIT  versus  off-equipment  ATS,  but  rather  how 
to  best  design  systems  with  integrated  diagnostics  capabilities  that  use  the  best  attributes  of 
both  to  sustain  high  system  availability  with  the  lowest  maintenance  resource  burden 
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(costs,  people,  spares,  space,  training,  manuals,  etc.).  There  is  no  indication  from  the 
information  obtained  in  this  study  that  either  BIT  or  off-equipment  ATS  will  have  a 
tendency  to  replace  the  other;  instead,  a  trend  towards  a  growing  interdependence  of  one 
technology  on  the  other  to  meet  mission  requirements  is  evident.  Further,  the  need  exists 
for  both  of  these  technologies  at  all  maintenance  levels,  with  possible  remote  processing 
and/or  communication  interface  capabilities  that  may  link  off-equipment  ATS  elements 
with  on-equipment  subsystem  I/O. 


9.6  DOD-UNIQUE  ATS  NEEDS 

When  examining  the  utilization  of  ATS  throughout  commercial  industry,  it  is  difficult 
to  find  another  user  of  ATS  with  the  same  field  mission  requirements  of  DoD.  The  closest 
comparison  for  the  fielded  ATS  came  from  the  airline  industry.  But  the  airline  industry, 
while  having  systems  of  similar  electronic  and  system  complexity,  has  products  that  are 
optimized  for  reliability  and  durability.  Mission  performance  parameters  tend  to  dominate 
DoD  systems  and  must  be  considered  often  ahead  of  reliability  and  durability  in  order  to 
fulfill  mission  objectives.  The  resulting  military  products  are  not  able  to  operate  like 
airliners  which  use  transported  spares  only  for  forward  maintenance  and  depot  maintenance 
for  lepairables.  While  the  trends  in  this  area  may  improve,  a  significant  change  in  DoD  field 
maintenance  requirements  is  not  anticipated  in  this  decade. 


Field-level  maintenance  deals  with  weapon  replaceable  assemblies  (WRA)  for  the  first 
level  of  off-equipment  maintenance  activities.  In  the  process  of  repairing  weapon 
replaceable  assemblies,  forward  maintenance  activities  generate  shop  repairable  units 
(SRU).  Forward  ATS  may  or  may  not  be  capable  of  testing  SRUs.  This  level  of  field  testing 
capability  varies  by  system  application  and  is  usually  a  function  of  several  operational 
factors:  mobility  requirements,  spares  availability  and  cost,  level  of  economical  repair, 
training,  facilities  and  space,  and  the  weapon  system/subsystem  reliability. 

DoD  depots  appeared  to  have  the  greatest  flexibility  in  choosing  specific  ATS.  The 
choices  for  depot  ATS  noted  during  this  study  were  generally  cost  based  relative  to 
immediate  needs.  There  were  examples  where  factory  equipment  was  transferred  to  depots. 
In  some  instances,  these  factory  tester  solutions  were  not  the  preferred  choice,  but  were 
temporary  solutions  due  to  schedule  or  cost  justifications.  Other  ATS  selection  examples 
ranged  from  COTS  products  to  sets  of  I-level  fielded  ATS. 
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